Positioning method for head-mounted display devices and head-mounted display devices

Abstract

This application relates to the field of SLAM technology, providing a localization method for a head-mounted display device and the head-mounted display device itself. To ensure sufficient feature extraction even in weakly textured scenes, a scale pyramid is constructed for each frame of images captured by a multi-view camera to extract corner and edge features, and line features are extracted for each frame. By fusing corner, edge, and line feature extraction algorithms, the number of feature points is effectively increased. Furthermore, to improve the accuracy of feature point matching, a feature point transformation matrix is ​​calculated based on the IMU poses of consecutive frames and the 3D coordinates of the extracted feature points. This matrix is ​​then used to track feature points on the scale pyramid of consecutive frames, ensuring continuous and stable tracking of the same feature point across different scales. This guarantees the stability and accuracy of feature point extraction, thereby effectively improving the stability and robustness of the SLAM algorithm.

Description

Technical Field

[0001] This application relates to the field of Simultaneous Localization and Mapping (SLAM) technology, and provides a positioning method for a head-mounted display device and the head-mounted display device itself. Background Technology

[0002] In recent years, with the introduction of the metaverse concept and the improvement of mobile computing capabilities, the technical requirements for users to fully interact in the fields of Virtual Reality (VR) and Augmented Reality (AR) have gradually increased. Therefore, key performance parameters such as the 6-degree-of-freedom (6DoF) positioning technology, tracking accuracy, and stability of the user's head on mobile platforms have become technical indicators that directly affect the user's actual interactive experience.

[0003] Currently, most head-mounted display (HMD) devices employ SLAM technology for head localization. In SLAM technology, the continuous and stable output of highly accurate and robust feature point matching pairs plays a crucial role in building dense and accurate maps. However, HMDs are mostly used in indoor spaces, which inevitably contain weak texture scenarios such as corners or repetitive patterns, like white walls, repetitive wallpaper, or empty corridors. It is difficult to extract enough accurate feature point matching pairs, resulting in inaccurate head localization and affecting the interactive experience. Summary of the Invention

[0004] This application provides a positioning method for a head-mounted display device and a head-mounted display device, which improves the accuracy and robustness of head positioning.

[0005] In a first aspect, embodiments of this application provide a positioning method for a head-mounted display device, comprising:

[0006] Acquire an image sequence captured by a multi-view camera, and construct a scale pyramid of the current frame image in the image sequence; the scale pyramid contains images of different scales generated after multiple downsampling of the current frame image;

[0007] The 3D coordinates of the first feature point corresponding to the previous frame image are obtained. The feature point transformation matrix between the previous frame image and the current frame image is determined by combining the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image. The first feature point is the feature point extracted from each layer of the scale pyramid of the previous frame image, and the feature point includes corner points and edge points.

[0008] Based on the feature point transformation matrix and the first feature point corresponding to the previous frame image, determine the second feature point that matches the first feature point in each layer of the scale pyramid of the current frame image;

[0009] Extract line features from the current frame image;

[0010] Based on the line features and second feature points corresponding to the current frame image, and the line features and first feature points corresponding to the previous frame image, a 3D-2D point matching pair is obtained;

[0011] The initial head pose corresponding to the current frame image is optimized based on the 3D-2D point matching pair to obtain the target head pose; the initial head pose is determined based on the second IMU pose.

[0012] The beneficial effects of the above technical solution are as follows: It enables head-mounted display devices to extract sufficient feature points even in weakly textured scenes. A scale pyramid is constructed for each frame of image captured by the multi-view camera, and corner and edge features are extracted from each layer of the scale pyramid. Line features are also extracted from each frame. By fusing corner, edge, and line feature extraction algorithms, the number of feature points in weakly textured scenes is effectively increased. Furthermore, to further improve the accuracy of feature point matching and ensure stable and robust algorithm operation, a feature point transformation matrix is ​​calculated based on the IMU poses of consecutive frames and the 3D coordinates of the extracted feature points. This matrix is ​​then used to track feature points in the scale pyramid of consecutive frames, ensuring continuous and stable tracking of the same feature point across different scales, thus guaranteeing the stability and accuracy of feature point extraction. In this way, when optimizing head pose based on the 3D-2D point matching pairs composed of extracted feature points and line features, the stability and robustness of the SLAM algorithm are effectively improved, thereby ensuring the accuracy and robustness of the localization results.

[0013] Optionally, obtaining the 3D coordinates of the first feature point corresponding to the previous frame image, and combining the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image to determine the feature point transformation matrix between the previous frame image and the current frame image includes:

[0014] The first camera pose is determined based on the first IMU pose corresponding to the previous frame image and the pre-calibrated camera extrinsic parameters, and the second camera pose is determined based on the second IMU pose corresponding to the current frame image and the camera extrinsic parameters.

[0015] Determine the pose transformation matrix based on the first camera pose and the second camera pose;

[0016] The feature point transformation matrix is ​​determined based on the pose transformation matrix and the 3D coordinates of the first feature point corresponding to the previous frame image.

[0017] The beneficial effects of the above technical solution are as follows: Since the IMU frame rate of the head-mounted display device is higher than that of the camera frame rate, and the IMU and camera have been calibrated at the factory, the pose transformation matrix between two consecutive frames can be determined based on the IMU pose and camera extrinsic parameters. Then, combined with the 3D coordinates of the extracted feature points, the feature point transformation matrix between two consecutive frames can be determined. Thus, feature point tracking can be performed on images of different scales using the feature point transformation matrix, thereby improving the stability and robustness of the localization algorithm.

[0018] Optionally, determining the feature point transformation matrix based on the pose transformation matrix and the 3D coordinates of the first feature point corresponding to the previous frame image includes:

[0019] The first feature point corresponding to the previous frame image is transformed to the base layer of the scale pyramid and decomposed along the two axes of the image coordinate system to obtain the first decomposition result; the base layer is the level of the scale pyramid where the previous frame image is located.

[0020] Based on the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after it has been transformed to the base layer, the 3D coordinates of the first feature point in the coordinate system of the previous frame are determined.

[0021] The 3D coordinates of the first feature point in the current frame coordinate system are transformed to the image coordinate system, and decomposed along the two axes of the image coordinate system to obtain the second decomposition result;

[0022] The feature point transformation matrix is ​​determined based on the first decomposition result and the second decomposition result.

[0023] The beneficial effects of the above scheme are as follows: Based on the pose transformation matrix and the first feature points extracted from each layer of the scale pyramid corresponding to the previous frame image, the 3D coordinates of the first feature point in the current coordinate system are determined. The movement of the feature point in the two frames can be decomposed along the two axes of the image coordinate system. Thus, the feature point transformation matrix is ​​determined using the decomposition results of the feature point in the two frames. In this way, the feature point can be tracked on images of different scales according to the feature point transformation matrix, ensuring that the extracted feature points are stable and have strong feature representation capabilities, thereby improving the stability and robustness of the overall localization algorithm.

[0024] Optionally, determining the 3D coordinates of the first feature point in the previous frame coordinate system based on the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after transformation to the base layer includes:

[0025] Based on the camera's internal parameters, the 2D coordinates of the first feature point corresponding to the previous frame image are converted into 3D coordinates on the normalized plane;

[0026] The 3D coordinates of the first feature point are multiplied by the corresponding components of the 3D coordinates after normalization to obtain the 3D coordinates of the first feature point in the coordinate system of the previous frame.

[0027] The beneficial effects of the above technical solution are as follows: Considering that visual odometry has cumulative errors, the 3D coordinates of the first feature point obtained from the SLAM backend are not very accurate. Therefore, instead of directly using the obtained 3D coordinates of the first feature point, it is multiplied by the 3D coordinates on the normalized plane to determine the 3D coordinates of the first feature point in the coordinate system of the previous frame. Compared with directly using the obtained 3D coordinates, the accuracy of the 3D coordinates of the feature point can be improved, thereby improving the accuracy of feature point tracking and ensuring the accuracy of the localization algorithm.

[0028] Optionally, determining the second feature point matching the first feature point in each layer of the scale pyramid of the current frame image based on the feature point transformation matrix and the first feature point corresponding to the previous frame image includes:

[0029] Calculate the determinant of the feature point transformation matrix, where the determinant value represents the degree of image scaling;

[0030] For the candidate layer images of the scale pyramid of the previous frame image, perform the following operations, wherein the candidate layer images are any layer images in the scale pyramid of the previous frame image:

[0031] Based on the value of the determinant, determine the target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image;

[0032] The first feature points extracted from the candidate layer image are matched with the feature points of the image corresponding to the target layer to obtain the second feature points in the image corresponding to the target layer.

[0033] The beneficial effects of the above technical solution are as follows: Since the feature point transformation matrix is ​​calculated at the base layer of the scale pyramid, for the first feature point in each layer of the scale pyramid of the previous frame image, it is necessary to determine which layer of the scale pyramid of the current frame image it appears in based on the determinant value representing the degree of image scaling. Thus, feature point extraction is performed on that layer of the image, achieving continuous and accurate tracking of feature points.

[0034] Optionally, determining the target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image based on the determinant value includes:

[0035] Starting from the base of the scale pyramid of the current frame image, perform the following operations on each traversal:

[0036] If the current level corresponding to the current determinant value is less than or equal to the number of levels in the scale pyramid, and the current determinant value does not exceed the scaling factor between two levels of the scale pyramid, then the current level is taken as the target level. Otherwise, the determinant value is reduced according to the scaling factor of the scale pyramid to traverse the next level of the image.

[0037] The beneficial effects of the above technical solution are as follows: when tracking feature points on images of different scales, the scale pyramid can be searched layer by layer by changing the determinant value of the feature point transformation matrix to determine the optimal level where the feature point is located, thereby extracting feature points on the image at that level, improving the accuracy of feature point matching on images of different scales, realizing continuous and accurate tracking of feature points, and thus improving the accuracy and robustness of the localization algorithm.

[0038] Optionally, extracting line features from the current frame image includes:

[0039] Obtain a pre-generated mask image with the same pixel size as the current frame image. The mask image contains a non-interest circular region with the image center as the origin and a preset number of pixels as the radius. The preset number of pixels is less than the number of pixels on the shorter side of the current frame image.

[0040] Line features are extracted from the edge regions of the current frame image based on the mask image.

[0041] The beneficial effects of the above technical solution are as follows: considering that line feature extraction is relatively time-consuming, a mask image is used to extract line features from the edge regions of each frame of the image, thereby ensuring the real-time performance of the localization algorithm.

[0042] Secondly, embodiments of this application provide a head-mounted display device, including a processor, a memory, a multi-view camera, and an IMU, wherein the IMU, the multi-view camera, the memory, and the processor are connected via a bus;

[0043] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0044] Acquire an image sequence captured by a multi-view camera, and construct a scale pyramid of the current frame image in the image sequence; the scale pyramid contains images of different scales generated after multiple downsampling of the current frame image;

[0045] The 3D coordinates of the first feature point corresponding to the previous frame image are obtained. Combined with the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image, a feature point transformation matrix between the previous frame image and the current frame image is determined. The first feature point is a feature point extracted from each layer of the scale pyramid of the previous frame image, and the feature point includes corner points and edge points. The first IMU pose is calculated based on the data acquired by the IMU corresponding to the previous frame image, and the second IMU pose is calculated based on the data acquired by the IMU corresponding to the current frame image.

[0046] Based on the feature point transformation matrix and the first feature point corresponding to the previous frame image, determine the second feature point that matches the first feature point in each layer of the scale pyramid of the current frame image;

[0047] Extract line features from the current frame image;

[0048] Based on the line features and second feature points corresponding to the current frame image, and the line features and first feature points corresponding to the previous frame image, a 3D-2D point matching pair is obtained;

[0049] The initial head pose corresponding to the current frame image is optimized based on the 3D-2D point matching pair to obtain the target head pose; the initial head pose is determined based on the second IMU pose.

[0050] Optionally, the processor obtains the 3D coordinates of the first feature point corresponding to the previous frame image, and combines the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image to determine the feature point transformation matrix between the previous frame image and the current frame image. Specifically, the operation is as follows:

[0051] The first camera pose is determined based on the first IMU pose corresponding to the previous frame image and the pre-calibrated camera extrinsic parameters, and the second camera pose is determined based on the second IMU pose corresponding to the current frame image and the camera extrinsic parameters.

[0052] Determine the pose transformation matrix based on the first camera pose and the second camera pose;

[0053] The feature point transformation matrix is ​​determined based on the pose transformation matrix and the 3D coordinates of the first feature point corresponding to the previous frame image.

[0054] Optionally, the processor determines the feature point transformation matrix based on the pose transformation matrix and the 3D coordinates of the first feature point corresponding to the previous frame image. Specifically, the operation is as follows:

[0055] The first feature point corresponding to the previous frame image is transformed to the base layer of the scale pyramid and decomposed along the two axes of the image coordinate system to obtain the first decomposition result; the base layer is the level of the scale pyramid where the previous frame image is located.

[0056] Based on the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after it has been transformed to the base layer, the 3D coordinates of the first feature point in the coordinate system of the previous frame are determined.

[0057] Based on the pose transformation matrix and the 3D coordinates of the first feature point in the coordinate system of the previous frame, determine the 3D coordinates of the first feature point in the coordinate system of the current frame.

[0058] The 3D coordinates of the first feature point in the current frame coordinate system are transformed to the image coordinate system, and decomposed along the two axes of the image coordinate system to obtain the second decomposition result;

[0059] The feature point transformation matrix is ​​determined based on the first decomposition result and the second decomposition result.

[0060] Optionally, the processor determines the 3D coordinates of the first feature point in the coordinate system of the previous frame based on the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after transformation to the base layer. Specifically, the operation is as follows:

[0061] Based on the camera's internal parameters, the 2D coordinates of the first feature point corresponding to the previous frame image are converted into 3D coordinates on the normalized plane;

[0062] The 3D coordinates of the first feature point are multiplied by the corresponding components of the 3D coordinates after normalization to obtain the 3D coordinates of the first feature point in the coordinate system of the previous frame.

[0063] Optionally, the processor determines, based on the feature point transformation matrix and the first feature point corresponding to the previous frame image, a second feature point matching the first feature point in each layer of the scale pyramid of the current frame image. Specifically, the operation is as follows:

[0064] Calculate the determinant of the feature point transformation matrix, where the determinant value represents the degree of image scaling;

[0065] For the candidate layer images of the scale pyramid of the previous frame image, perform the following operations, wherein the candidate layer images are any layer images in the scale pyramid of the previous frame image:

[0066] Based on the value of the determinant, determine the target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image;

[0067] The first feature points extracted from the candidate layer image are matched with the feature points of the image corresponding to the target layer to obtain the second feature points in the image corresponding to the target layer.

[0068] Optionally, the processor determines the target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image based on the determinant value, specifically by:

[0069] Starting from the base of the scale pyramid of the current frame image, perform the following operations on each traversal:

[0070] If the current level corresponding to the current determinant value is less than or equal to the number of levels in the scale pyramid, and the current determinant value does not exceed the scaling factor between two levels of the scale pyramid, then the current level is taken as the target level. Otherwise, the determinant value is reduced according to the scaling factor of the scale pyramid to traverse the next level of the image.

[0071] Optionally, the processor extracts line features from the current frame image, specifically by:

[0072] Obtain a pre-generated mask image with the same pixel size as the current frame image. The mask image contains a non-interest circular region with the image center as the origin and a preset number of pixels as the radius. The preset number of pixels is less than the number of pixels on the shorter side of the current frame image.

[0073] Line features are extracted from the edge regions of the current frame image based on the mask image.

[0074] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a head-mounted display device to perform any of the positioning methods for a head-mounted display device provided in embodiments of this application.

[0075] The technical effects of any of the implementation methods in the second or third aspect can be found in the technical effects of the corresponding implementation methods in the first aspect, and will not be repeated here. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1This is a schematic diagram of a head-mounted display device provided in an embodiment of this application;

[0078] Figure 2 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;

[0079] Figure 3 A flowchart illustrating the positioning method for a head-mounted display device provided in an embodiment of this application;

[0080] Figure 4 A schematic diagram of the scale pyramid constructed for an embodiment of this application;

[0081] Figure 5 This is a schematic diagram of fast corner feature extraction;

[0082] Figure 6 This is a schematic diagram of edge point feature extraction;

[0083] Figure 7 Flowchart of the method for determining the feature point transformation matrix;

[0084] Figure 8 A flowchart illustrating a method for feature point tracking on images at different scales;

[0085] Figure 9 Schematic diagram of feature point tracking effect on images at different scales;

[0086] Figure 10 This is a schematic diagram of a mask image;

[0087] Figure 11 A schematic diagram illustrating the extraction of corner points, edge points, and line features provided in an embodiment of this application;

[0088] Figure 12 This is an overall framework diagram of the SLAM localization algorithm provided in the embodiments of this application;

[0089] Figure 13 This is a structural diagram of the positioning device for a head-mounted display device provided in an embodiment of this application;

[0090] Figure 14 This is a structural diagram of a head-mounted display device provided in an embodiment of this application. Detailed Implementation

[0091] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0092] Based on the exemplary embodiments shown in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete technical solution on its own.

[0093] Furthermore, the terms “including” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0094] As used in this application, the term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0095] The terms used in the embodiments of this application are explained below.

[0096] IMU pose: 6DoF pose calculated using data acquired by the IMU, including translation and rotation.

[0097] The design concept of the embodiments of this application will be summarized below in conjunction with application scenarios.

[0098] VR / AR technology provides users with an immersive gaming experience. The translation and rotation of the user's head in three-dimensional space are important components of interaction in the VR / AR field. Accurately and stably determining the translation and rotation of the user's head ensures normal interaction in VR / AR applications. SLAM technology needs to extract a sufficient number of robust feature points to find the correct correspondence. If enough robust feature points cannot be extracted in scenes with weak textures, it is difficult to ensure the accuracy and robustness of head tracking and positioning in such scenes, which will inevitably cause certain problems in the user experience, such as virtual scene jitter and drift, thus making it difficult to interact accurately with the virtual scene.

[0099] In view of this, this application provides a localization method for a head-mounted display device. For weakly textured scenes, a scale pyramid is constructed for each frame of images captured by a multi-view camera. Corner and edge features are extracted from each layer of the scale pyramid to obtain scene features at different scales. Line features are also extracted from each frame. By fusing corner, edge, and line feature extraction algorithms, the number of trackable and matchable points in weakly textured scenes is effectively increased, allowing for the addition of more features to the map and achieving dense mapping for robust head localization. To further improve the accuracy of matching points and ensure stable and robust algorithm operation, after obtaining the 3D coordinates of the extracted feature points, a feature point transformation matrix is ​​determined based on the first IMU pose corresponding to the previous frame and the second IMU pose corresponding to the current frame. The feature points in each layer of the scale pyramid of the current frame are then tracked according to the feature point transformation matrix, ensuring continuous tracking of the same feature point across different scales, guaranteeing the stability and accuracy of feature point extraction, and ultimately achieving accurate and robust head localization.

[0100] See Figure 1 This is a schematic diagram of a head-mounted display device provided in an embodiment of this application. The head-mounted display device can be a VR device, an AR device, or a MR device. Multiple cameras are externally mounted on the head-mounted display device to capture images of the user's surroundings. An inertial measurement unit (IMU) is internally mounted to collect head acceleration and angular velocity. The IMU's frame rate is higher than that of the multiple cameras, and both the IMU and the multiple cameras are calibrated before the device leaves the factory.

[0101] It should be noted that the embodiments of this application do not impose restrictive requirements on the structure of the head-mounted display device, and its multi-view camera can be either dual-view or quad-view.

[0102] See Figure 2 This is a schematic diagram of an application scenario provided by an embodiment of this application. In an indoor scene with a smooth table, a user wears a head-mounted display device to experience 3D. During the experience, the head-mounted display device implements a SLAM algorithm based on a visual-inertial odometry (VIO) system composed of multi-view cameras and an IMU to locate the head, and uses the localization results to complete the interaction.

[0103] It should be noted that, Figure 2 This is merely an example scenario. This application embodiment does not impose any limiting requirements on the application scenario. It can be applied to indoor or outdoor scenarios.

[0104] It should be noted that the localization algorithm in this application embodiment only considers the inter-frame feature matching of the multi-camera (i.e., between consecutive frames of each camera), and does not consider the intra-frame feature matching of the multi-camera (i.e., between the same frames of the multi-camera).

[0105] See Figure 3 The above is a flowchart of a positioning method for a head-mounted display device provided in an embodiment of this application. This process can be executed by the head-mounted display device and mainly includes the following steps:

[0106] S301: Acquire the image sequence captured by the multi-view camera and construct the scale pyramid of the current frame image in the image sequence.

[0107] Specifically, for each image sequence captured by a camera, the following steps are performed: First, each frame in the image sequence is preprocessed into a grayscale image of a uniform data type (e.g., float32). Then, the grayscale image is scaled to obtain a scale pyramid containing the original image and the transformed image. The grayscale image of the original image serves as the base image of the scale pyramid, denoted as the 0th layer image.

[0108] Taking the current frame image as an example, such as Figure 4 The diagram shows the constructed scale pyramid. Assuming the current frame image is preprocessed into a grayscale image with a resolution of 640*480*1 (pixels), this grayscale image is downsampled four times, with each downsampling operation representing half the image's width and height, resulting in a five-layer scale pyramid containing the original image. The higher the pyramid level, the smaller the image size and the lower the resolution.

[0109] S302: Obtain the 3D coordinates of the first feature point corresponding to the previous frame image, and combine the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image to determine the feature point transformation matrix between the previous frame image and the current frame image.

[0110] In the embodiments of this application, for each frame of image scale pyramid constructed, feature points are extracted from each layer of the pyramid image, and the image layer and coordinates of each extracted feature point are recorded.

[0111] Optionally, feature point extraction for each image layer includes corner feature extraction and edge feature extraction.

[0112] In one example, the corner feature extraction algorithm uses the fast corner extraction algorithm. Since the fast corner extraction algorithm can detect areas with significant changes in local pixel grayscale, it is known for its fast extraction speed while ensuring a certain extraction accuracy. Therefore, the fast corner extraction algorithm is chosen to extract corner features.

[0113] like Figure 5The diagram illustrates fast corner extraction, which performs the following operation for each pixel in each layer of the scale pyramid: For pixel p in the image, obtain its grayscale value I. D Centered on pixel p, select 16 pixels on a circle with a radius of 3 pixels, and set the threshold T = 20%I. p If among the selected 16 points, 12 consecutive pixels have a brightness greater than I p +T or less than I D If -T is selected, then pixel p is considered a feature corner point.

[0114] In one example, edge point features can be extracted based on the gradient information of the grayscale image. Specifically, this is performed for each layer of the scale pyramid:

[0115] (1) Calculate the first derivative of the image Img and apply Gaussian blur to the image.

[0116] (2) Use the edge detection operator to calculate the first derivative of the image in the x and y directions, and obtain the gradient of each pixel in the x and y directions.

[0117] Edge detection operators include, but are not limited to, the Scharr operator, the Sobel operator, and the Canny operator. Taking the Scharr operator as an example, its convolution kernel is represented as:

[0118]

[0119] At this point, the gradient of each pixel (x1, y1) is:

[0120] grad x1 =(-3*Img(x1-1,y1-1)+3*Img(x1+1,y1-1)-10*Img(x1-1,y1)+10*Img(x1+1,y1)-3*Img(x1-1,y1+1)+3*Img(x1+1,y1+1)) / 3;

[0121] grad y1 =(-3*Img(x1-1, y1-1)-10*Img(x1, y1-1)+3*Img(x1+1, y1-1)-3*Img(x1-1, y1+1)+10*Img(x1, y1+1)+3*Img(x1+1, y1+1)) / 3.

[0122] (3) Based on the gradient of each pixel in the x and y directions, determine the angle and magnitude of the corresponding pixel. The calculation formula is as follows:

[0123]

[0124] in, The unit is radians, an g The unit angle is le1. The copy of each pixel can be recorded using a table of the same size and resolution as the image (Img).

[0125] (4) Filter out edge points based on the magnitude of each pixel.

[0126] Specifically, select the pixels around the pixel and compare their amplitudes. If the amplitude of the pixel is the largest and the amplitude of the pixel is greater than a preset change threshold, then the pixel is determined to be an edge point.

[0127] like Figure 6 The diagram shows the 9-point comparison method, which compares the magnitude of the pixel with its surrounding pixels.

[0128] In one example, to facilitate the management and use of feature points, after extracting the corner and edge points from each layer of the scale pyramid image, the coordinates of each feature point can be recorded using a grid image. Since both corner and edge points are feature points, the coordinate recording method does not need to distinguish between them; they are recorded in the same way.

[0129] Specifically, a new image is created with the same size as the feature extraction image, but with all pixels having a value of 0 (e.g., for an image with a resolution of 640*480, the new image also has a resolution of 640*480). Based on the coordinates of the extracted feature points, the pixel values ​​at the same locations are changed to 1, resulting in a grid image. This grid image serves as an intermediate result and can be used for subsequent feature matching and tracking processes. Compared to directly recording the coordinates of the feature points, it facilitates batch operations and feature recovery.

[0130] Based on the aforementioned methods for extracting corner and edge features, feature points are extracted for each frame of the image. The feature points extracted from each layer of the scale pyramid of the previous frame are designated as the first feature points, and the feature points extracted from each layer of the scale pyramid of the current frame are designated as the second feature points.

[0131] Each feature point can be identified by a unique ID, and the coordinates and image level of the corresponding feature point can be obtained through the ID.

[0132] Since the identifier of the same feature point is the same in images captured by a multi-camera system at different times, and the 3D coordinates of each feature point can be calculated by the SLAM backend after each feature point extraction, the 3D coordinates of the first feature point corresponding to the previous frame can be obtained from the SLAM backend based on the ID of the first feature point.

[0133] Considering that the scale of the features will change as the user moves (for example, if a box appears in front of the user and the user moves back and forth in front of the box, when the user gets closer to the box, the box tends to be an enlarged square feature, and when the user moves away from the box, the box tends to be a smaller point feature), in order to ensure the accuracy of feature point extraction, the tracking of feature points in the previous and next frame images in this application embodiment is continuous tracking under different scale images of the pyramid.

[0134] Since the IMU and multi-view camera in the head-mounted display device are calibrated at the factory, the camera's intrinsic and extrinsic parameters can be obtained through the calibration file. However, the IMU's acquisition frame rate is usually higher than that of the multi-view camera. Therefore, the pose determined by the IMU data, combined with the camera's intrinsic and extrinsic parameters, is used to determine the pose transformation matrix between consecutive frames. Then, based on the extracted feature points, the feature point transformation matrix is ​​determined to enable feature point tracking on images of different scales in consecutive frames, thereby improving the stability and robustness of the localization algorithm.

[0135] In one example, the process of determining the feature point transformation matrix is ​​as follows: Figure 7 As shown, it mainly includes the following steps:

[0136] S3021: Determine the first camera pose based on the first IMU pose corresponding to the previous frame image and the pre-calibrated camera extrinsic parameters, and determine the second camera pose based on the second IMU pose corresponding to the current frame image and the camera extrinsic parameters.

[0137] Assuming the time corresponding to the previous frame is t0 and the time corresponding to the current frame is t1, the first IMU pose calculated based on the IMU data at time t0 is: The second IMU pose calculated based on IMU data at time t1 is: The pre-calibrated external parameters from the IMU to the camera are Then the first camera pose at time t0 is:

[0138]

[0139] The pose of the second camera at time t1 is:

[0140]

[0141] S3022: Determine the pose transformation matrix based on the pose of the first camera and the pose of the second camera.

[0142] The pose transformation matrix between frames before and after time t0 and t1 is expressed as follows:

[0143]

[0144] S3023: Determine the feature point transformation matrix based on the pose transformation matrix and the 3D coordinates of the first feature point corresponding to the previous frame image.

[0145] For each layer of the scale pyramid of the previous frame image, the first feature point corresponding to the previous frame image can be obtained by using the feature extraction methods of corner points and edge points as described above. Through the SLAM backend, the 3D coordinates of each first feature point in the world coordinate system can be obtained. Then, combined with the pose transformation matrix, the feature point transformation matrix between the previous and next frames can be calculated.

[0146] It should be noted that since corner points and edge points are both point features, when calculating the feature point transformation matrix, corner points and edge points are no longer distinguished and are all treated as feature points.

[0147] Specifically, the calculation process of the feature point transformation matrix is ​​as follows:

[0148] S3023_1: Transform the first feature point corresponding to the previous frame image to the base layer of the scale pyramid, and decompose it along the two axes of the image coordinate system to obtain the first decomposition result.

[0149] The base layer is the 0th layer of the scale pyramid. The image at this layer retains the resolution of the original image. The images at other layers are the result of downsampling the image at this layer. Therefore, the coordinate transformation of feature points between images at different scales can be performed according to the scaling method between images when constructing the scale pyramid, and the image layer of the feature points remains unchanged.

[0150] Taking the scale pyramid downsampling ratio of 1 / 2 of the image width and height as an example, assuming that the coordinates of the feature point extracted from the nth layer (n=1,2,3...) of the scale pyramid image are (x,y), then the coordinates of the feature point transformed to the 0th layer image are (2n*x,2n*y).

[0151] As the user's head moves, the feature points also move. The movement vector of each feature point can be decomposed along the U-axis and V-axis of the image coordinate system. Assume that the first feature point extracted from each layer of the scale pyramid of the previous frame at time t0 moves p along the U-axis and V-axis respectively. du_t0 (du, 0) and p dv_t0 (0, dv).

[0152] S30232: Based on the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after it has been transformed to the base layer, determine the 3D coordinates of the first feature point in the coordinate system of the previous frame.

[0153] The process of determining the 3D coordinates of the first feature point in the coordinate system of the previous frame is as follows:

[0154] First, based on the camera's internal parameters, the 2D coordinates of the first feature point corresponding to the previous frame image are converted into 3D coordinates on the normalized plane.

[0155] Assuming the time corresponding to the previous frame is t0, let's take a first feature point p corresponding to time t0. t0 For example, suppose its 2D coordinates in the image are p t0 (u, v), then p t0 (u, v) can be converted to 3D coordinates on the normalized plane as follows: in, f x f y c x and c y For pre-calibrated camera intrinsic parameters, f x and f y c is the equivalent focal length of the camera. x and c y For the camera's optical center.

[0156] Then, the 3D coordinates of the first feature point are multiplied by the corresponding components of the 3D coordinates on the normalized plane to obtain the 3D coordinates of the first feature point in the coordinate system of the previous frame.

[0157] Assume the first feature point p is obtained from the SLAM backend t0 If the 3D coordinates are P0(x0, y0, z0), then the first feature point p t0 3D coordinates on the normalized plane Transform the 3D coordinates to the previous frame's coordinate system as P t0 (x t0 y t0 , z t0 ), where x t0 =x*z0,y t0 =y*z0, z t0 =z0.

[0158] In the embodiments of this application, considering that visual odometry has cumulative errors and the 3D coordinates of the first feature point obtained from the SLAM backend are not very accurate, the obtained 3D coordinates of the first feature point are not used directly. Instead, they are multiplied by the 2D coordinates on the image and the 3D coordinates on the normalized plane to determine the 3D coordinates of the first feature point in the coordinate system of the previous frame. Compared with directly using the obtained 3D coordinates, the accuracy of the 3D coordinates of the feature point can be improved, thereby improving the accuracy of feature point tracking and ensuring the accuracy of the localization algorithm.

[0159] S30233: Determine the 3D coordinates of the first feature point in the current frame coordinate system based on the pose transformation matrix and the 3D coordinates of the first feature point in the previous frame coordinate system.

[0160] Using a pose transformation matrix, the 3D coordinates of the first feature point in the previous frame coordinate system are transformed to the current frame coordinate system, thus obtaining the 3D coordinates P of the first feature point in the current frame coordinate system. t1 (x t1 y t1 , z t1 The formula is expressed as:

[0161]

[0162] S3023_4: Transform the 3D coordinates of the first feature point in the current frame coordinate system to the image coordinate system, and decompose it along the two axes of the image coordinate system to obtain the second decomposition result.

[0163] Based on the pre-calibrated camera parameters, the 3D coordinates P of the first feature point in the current frame coordinate system can be determined. t1 (x t1 y t1 , z t1 Transform the image coordinates at time t1 to obtain the 2D coordinates p of the first feature point at time t1. t1 (u, v), the formula is expressed as:

[0164]

[0165] Similarly, by decomposing pt1(u, v) along the U-axis and V-axis of the image coordinate system, we can obtain the movement components of the first feature point at time t1, which are p du_t1 and p dv_t1 .

[0166] S30235: Determine the feature point transformation matrix based on the first decomposition result and the second decomposition result.

[0167] Suppose the transformation matrix of the first feature point extracted at time t0 to the feature point at time t1 is represented as follows: Then p at time t0 du_t0 The formula for transforming to time t1 is expressed as follows: Right now:

[0168]

[0169] Similarly, p at time t0 dv_t0 Transform to time t1 and substitute p dv_t1 It can be solved

[0170] In the embodiments of this application, the 3D coordinates of the first feature point in the current coordinate system are determined based on the pose transformation matrix and the first feature points in each layer of the scale pyramid of the previous frame image. The movement of the feature point in the two frames can be decomposed along the two axes of the image coordinate system. The feature point transformation matrix is ​​then determined using the decomposition results of the feature point in the two frames. In this way, the feature point can be tracked in images at different scales based on the feature point transformation matrix, ensuring that the same feature point can be tracked continuously and stably in images at different scales, extending the "lifetime" of the extracted feature points, ensuring that the extracted feature points are stable and have strong feature representation capabilities, thereby improving the stability and robustness of the overall localization algorithm.

[0171] S303: Based on the feature point transformation matrix and the first feature point corresponding to the previous frame image, determine the second feature point that matches the first feature point in each layer of the scale pyramid of the current frame image.

[0172] Since the solution to the feature point transformation matrix is ​​performed at level 0 of the scale pyramid, to track feature points on images at different scales, it is necessary to determine which level of the first feature point extracted at time t0 (i.e., the scale pyramid of the previous frame) appears at time t1 (i.e., the scale pyramid of the current frame). The determinant of the feature point transformation matrix represents the scaling degree between different levels of the scale pyramid. Therefore, based on the determinant of the feature point transformation matrix, the optimal tracking level of the first feature point extracted from the scale pyramid of the previous frame in the scale pyramid of the current frame can be determined. Thus, for each level of the scale pyramid of the previous frame, using the image of that level, the first feature point extracted from that level, and the image of the corresponding level of the first feature point in the scale pyramid of the current frame as input, and employing optical flow matching, feature points can be accurately tracked at the levels of the scale pyramid of the current frame, achieving feature point tracking on images at different scales.

[0173] For details on the feature point tracking process, please refer to [link / reference]. Figure 8 It mainly includes the following steps:

[0174] S3031: Calculate the determinant of the feature point transformation matrix.

[0175] As the user's head moves, the scale of the features also changes. To determine the scale after the feature changes, the determinant of the feature point transformation matrix can be calculated.

[0176] Since the calculation of the feature point transformation matrix is ​​performed at level 0 of the scale pyramid, the values ​​of the rows and columns at this point represent the scale of the original image, i.e., before scaling.

[0177] S3032: Based on the determinant value, determine the target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image; wherein, the candidate layer image is any layer image in the scale pyramid of the previous frame image.

[0178] To achieve tracking across images at different scales, for the first feature point extracted from the candidate layer image, it is necessary to determine the target level of each first feature point in the scale pyramid of the current frame image. Specifically, the target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image can be determined by traversing from the base layer (i.e., layer 0) of the scale pyramid. Each traversal involves the following steps:

[0179] S3032_1: Determine whether the current determinant value is less than or equal to the current level in the scale pyramid of the current frame image. If yes, execute S30322; otherwise, execute S30325.

[0180] During the traversal, it is necessary to ensure that the process is performed within the scale pyramid of the current frame image. Therefore, it is necessary to ensure that the current level is less than or equal to the number of levels in the scale pyramid.

[0181] S30322: Determine whether the current determinant value does not exceed the scaling factor between the two levels of the scale pyramid. If yes, execute S30323; otherwise, execute S30324.

[0182] Assuming the scaling factor between two adjacent layers in the scale pyramid is 2, if the current determinant value exceeds 3, it is equivalent to skipping layers for a search. Therefore, it is necessary to ensure that the current determinant value does not exceed the scaling factor between two layers of the scale pyramid.

[0183] S30323: Set the current level as the target level and end the traversal.

[0184] S30324: Reduce the determinant value according to the scaling factor of the scale pyramid to traverse the next level of the image.

[0185] When the determinant value exceeds the scaling factor between two levels of the scale pyramid and the current level does not exceed the number of levels in the scale pyramid, the optimal target level is determined by progressively reducing D. Each time the determinant value is reduced, the number of image levels traversed increases by one.

[0186] Assuming that during the construction of the scale pyramid, the image is shrunk by 1 / 2 in the U and V directions during each downsampling process, the determinant of the feature point transformation matrix in each layer of the image is shrunk by (1 / 2)*(1 / 2) each time, that is, the determinant is shrunk by 0.25 each time.

[0187] S30325: Use the base level of the scale pyramid as the target level.

[0188] After the traversal is complete, if no level is found whose determinant value does not exceed the scaling factor between two levels of the scale pyramid and is less than or equal to the scale pyramid level, the base level (i.e., level 0) is taken as the target level.

[0189] In the embodiments of this application, since the feature point transformation matrix is ​​calculated at the base level of the scale pyramid, when tracking feature points on images at different scales, the scale pyramid can be searched layer by layer by changing the determinant value of the feature point transformation matrix to determine the optimal level where the feature point is located. This allows for the extraction of feature points on that layer of the image, improving the accuracy of feature point matching on images at different scales, enabling continuous and accurate tracking of feature points, and thus improving the accuracy and robustness of the localization algorithm.

[0190] S3033: Perform feature point matching between the first feature point extracted from the candidate layer image and the image corresponding to the target layer to obtain the second feature point in the image corresponding to the target layer.

[0191] In one example, using the aforementioned feature extraction method for corner and edge points, the second feature point in the image corresponding to the target level in the scale pyramid of the current frame image is extracted, and the descriptor of the second feature point extracted from the image corresponding to the target level and the descriptor of the first feature point extracted from the candidate layer image are calculated. Based on the descriptor, the feature point that matches the first feature point among the currently extracted second feature points is determined.

[0192] like Figure 9 As shown, this is a schematic diagram of the feature point tracking effect on images at different scales. The first feature point extracted from the 0th layer of the scale pyramid of the previous frame image is matched with the second feature point extracted from the 2nd layer of the scale pyramid of the current frame image.

[0193] In the embodiments of this application, the determinant of the feature point transformation matrix is ​​used to determine the level of the scale pyramid, thereby realizing the tracking of feature points on images at different scales, ensuring that feature point feature changes caused by movement changes between consecutive frames are not easily lost, and improving the robustness of inter-frame feature point matching.

[0194] S304: Extract line features from the current frame image.

[0195] To enhance the accuracy of the localization algorithm by adding more features, line feature extraction was performed in addition to feature point matching on images at different scales. Unlike feature point extraction, considering that line feature extraction is time-consuming, this application performed line feature extraction on each frame of images acquired by the multi-view camera.

[0196] In one example, considering the high requirements for the response speed of the head-mounted display device during the user handover process, in order to ensure the real-time performance of the positioning algorithm, only the edge regions of each frame of image can be extracted for line features.

[0197] Taking the line feature extraction process of the current frame image as an example, a pre-generated mask image with the same pixel size as the current frame image is obtained. Line features are then extracted from the edge regions of the current frame image based on the mask image. The mask image contains a circular region with the image center as the origin and a preset number of pixels as the radius. This circular region is considered a region of no interest, and the preset number of pixels is less than the number of pixels on the shorter side of the current frame image, thus preserving the edge regions. Line features are extracted only from the edge regions.

[0198] like Figure 10 As shown, this is a schematic diagram of a mask image. Assuming the resolution of the current frame image is 640*480 (pixels), the size of the mask image is also 640*480 (pixels), and the radius of the circular area in the center is 180 pixels.

[0199] In practice, the process of extracting line features includes:

[0200] First, calculate the gradient of each pixel in the current frame image.

[0201] To record changes in brightness and darkness and obtain the position of line segment edges, the gradient of each pixel can be calculated using the following formula:

[0202]

[0203] Then, based on the gradient of each pixel in the current frame image, the angle and magnitude of the corresponding pixel are calculated using the following formula:

[0204]

[0205] As can be seen from Equations 11 and 1, the gradient direction of the line feature is perpendicular to the gradient direction of the edge point.

[0206] Secondly, based on the angle of each pixel, at least one straight line region is determined.

[0207] Since lines are made up of points, and each pixel has a direction (angle), at least one straight line region can be determined by calculating the difference between every two pixels.

[0208] Specifically, from the pixels within the edge region, a pixel is selected, and its angle is used as the initial angle of a straight line region. The angle differences between this pixel and its surrounding pixels are calculated. If the angle difference is less than a preset empirical value (e.g., 22°), the corresponding surrounding pixels can be included in the straight line region. The angle of the straight line region is updated once for each new pixel included, as follows:

[0209]

[0210] Where i represents the number of pixels included.

[0211] In one example, if the endpoint coordinates of a straight line region lie within a circular region of the mask image, that straight line region is discarded, thus enabling the extraction of line features only from the edge regions of the current frame image. By extracting line features only from the image edge regions, the localization algorithm maintains real-time performance and stability, while the inclusion of line features also makes the localization algorithm more robust.

[0212] Finally, a rectangle is calculated based on at least one straight-line region to obtain the line feature.

[0213] After obtaining at least one straight-line region, each straight-line region is considered as a rectangular rigid body. The magnitude of each pixel within the straight-line region is considered as the mass of that region. Therefore, the center of mass of each straight-line region corresponds to the center of a rectangle. The width and length of the rectangle are the minimum values ​​that allow the rectangle to cover the entire straight-line region. The direction of the rectangle is the updated angle. line The center of the rectangle is (centimeters). x , cen y The formula is expressed as follows:

[0214]

[0215]

[0216] By calculating the rectangle as described above, line features can be extracted.

[0217] like Figure 11 The diagram shows the extraction of corner points, edge points, and line features provided in this application embodiment. By extracting corner point features and edge point features on images at different scales, and extracting line features in the acquired images, the problem of insufficient feature point extraction in weak texture scenes (such as areas with blurred spatial features such as a small room with only a few items or surrounded by white walls) can be overcome by traditional feature extraction algorithms, thus ensuring the robustness of localization and mapping in weak texture scenes.

[0218] S305: Obtain 3D-2D point matching pairs based on the line features and second feature points corresponding to the current frame image, and the line features and first feature points corresponding to the previous frame image.

[0219] The second feature point tracked in the scale pyramid of the current frame image is the same 3D point in three-dimensional space as the first feature point matched in the scale pyramid of the previous frame image. After the first feature point is extracted, the optimized 3D coordinates can be obtained through the SLAM backend. These 3D coordinates and the 2D coordinates of the second feature point can form 3D-2D point matching pairs. Line features can be regarded as being composed of individual pixel features. Therefore, the line features in the current frame image and the line features in the previous frame image can be decomposed into point features. Combined with the 3D coordinates of the points contained in the line features obtained from the SLAM backend, they can also form 3D-2D point matching pairs.

[0220] S306: Optimize the initial head pose corresponding to the current frame image based on the 3D-2D point matching pair to obtain the target head pose.

[0221] The initial head pose of the current frame image acquired at time t1 can be determined by calculating the second IMU pose based on the data collected by the IMU at time t1 and the pre-calibrated camera extrinsic parameters. The SALM backend then uses the 3D-2D point matching pairs determined by S305 to optimize the pose and obtain the target head pose.

[0222] See Figure 12 This is a general framework diagram of the SLAM localization algorithm provided in this application embodiment, mainly including data acquisition, front-end estimation, back-end optimization, loop closure detection, and mapping. The data acquisition part mainly uses a camera to acquire images; the front-end estimation part is used for feature extraction and to provide initial pose values. In the feature extraction process, a scale pyramid is constructed for each frame of image. Corner and edge features are extracted in the scale pyramid, and the extracted feature points are tracked on images at different scales in consecutive frames. Line features are extracted at the 0th layer of the scale pyramid, and 3D-2D matching point pairs are generated based on the extracted line features and feature points; the back-end optimization part mainly uses the 3D-2D matching point pairs and loop closure detection information to optimize the pose; the mapping part mainly uses the optimized pose trajectory to construct a map.

[0223] In the embodiments of this application, to ensure sufficient feature points can be extracted for head-mounted display devices even in weakly textured scenes, a scale pyramid is constructed for each frame of image captured by a multi-view camera. Corner and edge features are extracted from each layer of the scale pyramid, and line features are extracted from each frame. By fusing corner, edge, and line feature extraction algorithms, the number of feature points in weakly textured scenes is effectively increased. Furthermore, to further improve the accuracy of feature point matching and enable the algorithm to run stably and robustly, a feature point transformation matrix is ​​calculated based on the IMU poses of consecutive frames and the 3D coordinates of the extracted feature points. This matrix is ​​then used to track feature points in the scale pyramids of consecutive frames, ensuring continuous and stable tracking of the same feature point across different scales, thus guaranteeing the stability and accuracy of feature point extraction. In this way, when optimizing head pose based on the 3D-2D point matching pairs composed of extracted feature points and line features, the stability and robustness of the SLAM algorithm are effectively improved, ensuring the accuracy and robustness of the localization results and providing a better interactive experience for the wearer of the head-mounted display device.

[0224] Based on the same technical concept, embodiments of this application provide a positioning device for a head-mounted display device, the structure of which is as follows: Figure 13 As shown, it includes a pyramid construction module 1301, a transformation matrix calculation module 1302, a feature extraction module 1303, and a pose optimization module 1304, wherein:

[0225] The pyramid construction module 1301 is used to acquire image sequences captured by multi-view cameras and construct a scale pyramid of the current frame image in the image sequence; the scale pyramid contains images of different scales generated after multiple downsampling of the current frame image;

[0226] The transformation matrix calculation module 1302 is used to obtain the 3D coordinates of the first feature point corresponding to the previous frame image, and combine the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image to determine the feature point transformation matrix between the previous frame image and the current frame image; wherein, the first feature point is the feature point extracted from each layer of the scale pyramid of the previous frame image, and the feature point includes corner points and edge points.

[0227] The feature extraction module 1303 is used to determine the second feature point matched by the first feature point in each layer of the scale pyramid of the current frame image based on the feature point transformation matrix and the first feature point corresponding to the previous frame image; and to extract line features from the current frame image and obtain 3D-2D point matching pairs based on the line features and second feature points corresponding to the current frame image, as well as the line features and first feature points corresponding to the previous frame image.

[0228] The pose optimization module 1304 is used to optimize the initial head pose corresponding to the current frame image based on the 3D-2D point matching pair to obtain the target head pose; the initial head pose is determined based on the second IMU pose.

[0229] Optionally, the transformation matrix calculation module 1302 is specifically used for:

[0230] The first camera pose is determined based on the first IMU pose corresponding to the previous frame image and the pre-calibrated camera extrinsic parameters, and the second camera pose is determined based on the second IMU pose corresponding to the current frame image and the camera extrinsic parameters.

[0231] Determine the pose transformation matrix based on the poses of the first and second cameras;

[0232] The feature point transformation matrix is ​​determined based on the pose transformation matrix and the 3D coordinates of the first feature point corresponding to the previous frame image.

[0233] Optionally, the transformation matrix calculation module 1302 is specifically used for:

[0234] The first feature point corresponding to the previous frame image is transformed to the base layer of the scale pyramid and decomposed along the two axes of the image coordinate system to obtain the first decomposition result; the base layer is the level of the scale pyramid where the previous frame image is located.

[0235] Based on the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after it has been transformed to the base layer, the 3D coordinates of the first feature point in the coordinate system of the previous frame are determined.

[0236] Based on the pose transformation matrix and the 3D coordinates of the first feature point in the coordinate system of the previous frame, determine the 3D coordinates of the first feature point in the coordinate system of the current frame.

[0237] The 3D coordinates of the first feature point in the current frame coordinate system are transformed to the image coordinate system, and then decomposed along the two axes of the image coordinate system to obtain the second decomposition result;

[0238] Based on the first and second decomposition results, the feature point transformation matrix is ​​determined.

[0239] Optionally, the transformation matrix calculation module 1302 is specifically used for:

[0240] Based on the camera's internal parameters, the 2D coordinates of the first feature point corresponding to the previous frame image are converted into 3D coordinates on the normalized plane;

[0241] Multiply the 3D coordinates of the first feature point obtained by the corresponding component of the 3D coordinates after normalization on the plane to obtain the 3D coordinates of the first feature point in the coordinate system of the previous frame.

[0242] Optional, feature extraction module 1303, specifically used for:

[0243] Calculate the determinant of the feature point transformation matrix, where the determinant value represents the degree of image scaling.

[0244] For the candidate layer images of the scale pyramid of the previous frame, perform the following operations, where the candidate layer images are any layer images in the scale pyramid of the previous frame:

[0245] Based on the value of the determinant, determine the target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image;

[0246] The first feature point extracted from the candidate layer image is matched with the feature point of the image corresponding to the target layer to obtain the second feature point in the image corresponding to the target layer.

[0247] Optional, feature extraction module 1303, specifically used for:

[0248] Starting from the base of the scale pyramid of the current frame image, perform the following operations on each iteration:

[0249] If the current determinant value corresponds to a current level that is less than or equal to the number of levels in the scale pyramid, and the current determinant value does not exceed the scaling factor between two levels of the scale pyramid, then the current level is taken as the target level. Otherwise, the determinant value is reduced according to the scaling factor of the scale pyramid to traverse the next level of the image.

[0250] For ease of description, the head-mounted display device can be divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware.

[0251] Based on the same technical concept, this application provides a head-mounted display device that can implement the steps of the positioning method of the head-mounted display device described above, and can achieve the same technical effect.

[0252] See Figure 14 The head-mounted display device includes a processor 1401, a memory 1402, a multi-view camera 1403, and an IMU 1404, which are connected via a bus 1405.

[0253] The IMU 1404 is used to provide data for calculating the IMU pose;

[0254] The multi-view camera 1403 is used to acquire images;

[0255] The memory 1002 stores a computer program, and the processor 1001 executes the steps of the positioning method for the head-mounted display device provided in the embodiments of this application according to the computer program.

[0256] In this embodiment, the memory 1402 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and programs required for running instant messaging functions; the data storage area may store various instant messaging information and operation instruction sets. The memory 1402 may be volatile memory, such as random-access memory (RAM); it may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it may be any other medium capable of carrying or storing a desired computer program having an instruction or data structure form and accessible by a computer, but is not limited thereto. The memory 1402 may be a combination of the above-described memories.

[0257] The processor 1401 may include one or more central processing units (CPUs), GPUs, or digital processing units, etc. The processor 1401 is used to implement the steps of any of the above-described positioning methods for a head-mounted display device when it invokes a computer program stored in the memory 1402.

[0258] It should be noted that, Figure 14 This is merely an example illustrating the hardware necessary for a head-mounted display device to perform the positioning method steps provided in the embodiments of this application. Not shown, the head-mounted display device may also include conventional hardware such as a display screen, speakers, microphones, communication interfaces, power supplies, handles, and buttons.

[0259] This application embodiment does not limit the specific connection medium between the IMU 1404, multi-view camera 1403, memory 1402, and processor 1401. In this application embodiment, the bus 1405 between the IMU 1404, multi-view camera 1403, memory 1402, and processor 1401 is... Figure 14 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1405 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 14 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0260] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0261] This application also provides a computer-readable storage medium for storing instructions that, when executed, can complete the steps of any of the head-mounted display device positioning methods described in the foregoing embodiments.

[0262] This application also provides a computer program product for storing a computer program that performs the steps of any of the positioning methods for head-mounted display devices described in the foregoing embodiments.

[0263] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-RvM, optical storage, etc.) containing computer-usable program code.

[0264] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0265] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0266] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0267] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A positioning method for a head-mounted display device, characterized in that, include: Acquire image sequences captured by multi-view cameras and construct a scale pyramid for the current frame image in the image sequence; The scale pyramid contains images of different scales generated after multiple downsamplings of the current frame image; The 3D coordinates of the first feature point corresponding to the previous frame image are obtained. The feature point transformation matrix between the previous frame image and the current frame image is determined by combining the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image. The first feature point is the feature point extracted from each layer of the scale pyramid of the previous frame image, and the feature point includes corner points and edge points. Based on the feature point transformation matrix and the first feature point corresponding to the previous frame image, determine the second feature point that matches the first feature point in each layer of the scale pyramid of the current frame image; Extract line features from the current frame image; Based on the line features and second feature points corresponding to the current frame image, and the line features and first feature points corresponding to the previous frame image, a 3D-2D point matching pair is obtained; The initial head pose corresponding to the current frame image is optimized based on the 3D-2D point matching pair to obtain the target head pose; the initial head pose is determined based on the second IMU pose.

2. The method as described in claim 1, characterized in that, The step of obtaining the 3D coordinates of the first feature point corresponding to the previous frame image, and combining the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image to determine the feature point transformation matrix between the previous frame image and the current frame image includes: The first camera pose is determined based on the first IMU pose corresponding to the previous frame image and the pre-calibrated camera extrinsic parameters, and the second camera pose is determined based on the second IMU pose corresponding to the current frame image and the camera extrinsic parameters. Determine the pose transformation matrix based on the first camera pose and the second camera pose; The feature point transformation matrix is ​​determined based on the pose transformation matrix and the 3D coordinates of the first feature point corresponding to the previous frame image.

3. The method as described in claim 2, characterized in that, The step of determining the feature point transformation matrix based on the pose transformation matrix and the 3D coordinates of the first feature point corresponding to the previous frame image includes: The first feature point corresponding to the previous frame image is transformed to the base layer of the scale pyramid and decomposed along the two axes of the image coordinate system to obtain the first decomposition result; the base layer is the level of the scale pyramid where the previous frame image is located. Based on the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after it has been transformed to the base layer, the 3D coordinates of the first feature point in the coordinate system of the previous frame are determined. Based on the pose transformation matrix and the 3D coordinates of the first feature point in the coordinate system of the previous frame, determine the 3D coordinates of the first feature point in the coordinate system of the current frame. The 3D coordinates of the first feature point in the current frame coordinate system are transformed to the image coordinate system, and decomposed along the two axes of the image coordinate system to obtain the second decomposition result; The feature point transformation matrix is ​​determined based on the first decomposition result and the second decomposition result.

4. The method as described in claim 3, characterized in that, Determining the 3D coordinates of the first feature point in the previous frame coordinate system based on the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after transformation to the base layer includes: Based on the camera's internal parameters, the 2D coordinates of the first feature point corresponding to the previous frame image are converted into 3D coordinates on the normalized plane; The 3D coordinates of the first feature point are multiplied by the corresponding components of the 3D coordinates after normalization to obtain the 3D coordinates of the first feature point in the coordinate system of the previous frame.

5. The method as described in claim 1, characterized in that, The step of determining the second feature point that matches the first feature point in each layer of the scale pyramid of the current frame image based on the feature point transformation matrix and the first feature point corresponding to the previous frame image includes: Calculate the determinant of the feature point transformation matrix, where the determinant value represents the degree of image scaling; For the candidate layer images of the scale pyramid of the previous frame image, perform the following operations, wherein the candidate layer images are any layer images in the scale pyramid of the previous frame image: Based on the value of the determinant, determine the target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image; The first feature points extracted from the candidate layer image are matched with the feature points of the image corresponding to the target layer to obtain the second feature points in the image corresponding to the target layer.

6. The method as described in claim 5, characterized in that, The step of determining the target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image based on the determinant value includes: Starting from the base of the scale pyramid of the current frame image, perform the following operations on each traversal: If the current level corresponding to the current determinant value is less than or equal to the number of levels in the scale pyramid, and the current determinant value does not exceed the scaling factor between two levels of the scale pyramid, then the current level is taken as the target level. Otherwise, the determinant value is reduced according to the scaling factor of the scale pyramid to traverse the next level of the image.

7. The method as described in claim 1, characterized in that, Extracting line features from the current frame image includes: Obtain a pre-generated mask image with the same pixel size as the current frame image. The mask image contains a non-interest circular region with the image center as the origin and a preset number of pixels as the radius. The preset number of pixels is less than the number of pixels on the shorter side of the current frame image. Line features are extracted from the edge regions of the current frame image based on the mask image.

8. A head-mounted display device, characterized in that, It includes a processor, a memory, a multi-view camera, and an IMU, wherein the IMU, the multi-view camera, the memory, and the processor are connected via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: Acquire an image sequence captured by a multi-view camera, and construct a scale pyramid of the current frame image in the image sequence; the scale pyramid contains images of different scales generated after multiple downsampling of the current frame image; The 3D coordinates of the first feature point corresponding to the previous frame image are obtained. The feature point transformation matrix between the previous frame image and the current frame image is determined by combining the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image. The first feature point is the feature point extracted from each layer of the scale pyramid of the previous frame image, and the feature point includes corner points and edge points. Based on the feature point transformation matrix and the first feature point corresponding to the previous frame image, determine the second feature point that matches the first feature point in each layer of the scale pyramid of the current frame image; Extract line features from the current frame image; Based on the line features and second feature points corresponding to the current frame image, and the line features and first feature points corresponding to the previous frame image, a 3D-2D point matching pair is obtained; The initial head pose corresponding to the current frame image is optimized based on the 3D-2D point matching pair to obtain the target head pose; the initial head pose is determined based on the second IMU pose.

9. The head-mounted display device as described in claim 8, characterized in that, The processor extracts line features from the current frame image, specifically by: Obtain a pre-generated mask image with the same pixel size as the current frame image. The mask image contains a non-interest circular region with the image center as the origin and a preset number of pixels as the radius. The preset number of pixels is less than the number of pixels on the shorter side of the current frame image. Line features are extracted from the edge regions of the current frame image based on the mask image.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing the head-mounted display device to perform the method of any one of claims 1 to 7.

Images