Positioning method of head-mounted display device and head-mounted display device

Abstract

The invention relates to the technical field of SLAM (Simultaneous Localization and Mapping), provides a positioning method of a head-mounted display device and the head-mounted display device, and aims to construct a scale pyramid for each frame of image collected by a multi-view camera to perform feature extraction of angular points and edge points in order to ensure that enough features can be extracted in a weak texture scene. And carrying out line feature extraction on each frame of image, and effectively increasing the number of feature points by fusing angular points, edge points and a line feature extraction algorithm. On the other hand, in order to improve the accuracy of feature point matching, a feature point conversion matrix is calculated according to the IMU poses corresponding to the front and back frame images and the 3D coordinates of the extracted feature points, and feature point tracking is performed on the scale pyramids of the front and back frame images by using the feature point conversion matrix; therefore, it is ensured that the same feature point is continuously and stably tracked under the images of different scales, the stability and accuracy of feature point extraction are ensured, and the stability and robustness of the SLAM algorithm are effectively improved.

Description

Background Art

[0002] In recent years, with the introduction of the concept of the metaverse and the improvement of mobile computing capabilities, the market's technical requirements for users to fully interact in the field of virtual reality (VR) / augmented reality (AR) technology have gradually increased. Therefore, key performance parameters such as the six degrees of freedom (6DoF) positioning technology, tracking accuracy and stability of the user's head on mobile platforms have become technical indicators that directly affect the user's actual interactive experience.

[0003] At present, most head-mounted display devices (HMD) use SLAM technology for head positioning. In SLAM technology, the continuous and stable output of high-accuracy and robust feature point matching pairs plays a pivotal role in building dense and accurate maps. However, HMD is mostly used in indoor spaces, which inevitably have corners or weak textures with repeated patterns, such as white walls around, repeated wallpaper around, or empty corridors. It is impossible to extract enough and accurate feature point matching pairs, resulting in inaccurate head positioning and affecting the interactive experience. Summary of the invention

[0004] The embodiments of the present application provide a positioning method for a head-mounted display device and a head-mounted display device, which are used to improve the accuracy and robustness of head positioning.

[0005] In a first aspect, an embodiment of the present application provides a positioning method for a head mounted display device, comprising:

[0006] Acquire an image sequence collected by a multi-camera, and construct a scale pyramid of a current frame image in the image sequence; the scale pyramid includes images of different scales generated after multiple downsampling of the current frame image;

[0007] Obtain the 3D coordinates of the first feature point corresponding to the previous frame image, and determine the feature point conversion matrix between the previous frame image and the current frame image in combination with the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image; wherein the first feature point is a feature point extracted from each layer of the image included in the scale pyramid of the previous frame image, and the feature point includes a corner point and an edge point;

[0008] Determine, according to the feature point conversion matrix and the first feature point corresponding to the previous frame image, a second feature point matched by the first feature point in each layer image included in the scale pyramid of the current frame image;

[0009] Extracting line features from the current frame image;

[0010] Obtaining a 3D-2D point matching pair according to the line feature and the second feature point corresponding to the current frame image, and the line feature and the first feature point corresponding to the previous frame image;

[0011] The initial head pose corresponding to the current frame image is optimized according to the 3D-2D point matching pairs to obtain a target head pose; the initial head pose is determined according to the second IMU pose.

[0012] The beneficial effects of the above technical solution are: for the head-mounted display device, sufficient feature points can be extracted in weak texture scenes. A scale pyramid is constructed for each frame of images collected by the multi-eye camera, and corner point and edge point features are extracted from each layer of the scale pyramid, and line features are extracted for each frame of images. By fusing corner point, edge point and line feature extraction algorithms, the number of feature points in weak texture scenes is effectively increased. On the other hand, in order to further improve the accuracy of feature point matching and enable the algorithm to run stably and robustly, the feature point conversion matrix is ​​calculated according to the IMU pose corresponding to the previous and next frame images and the 3D coordinates of the extracted feature points, and the feature point conversion matrix is ​​used to track the feature points of the scale pyramid of the previous and next frame images, thereby ensuring the continuous and stable tracking of the same feature point under different scale images, and ensuring the stability and accuracy of feature point extraction. In this way, when optimizing the head pose according to the 3D-2D point matching pairs composed of the extracted feature points and line features, the stability and robustness of the SLAM algorithm can be effectively improved, thereby ensuring the accuracy and robustness of the positioning results.

[0013] Optionally, acquiring the 3D coordinates of the first feature point corresponding to the previous frame image, combining the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image, and determining the feature point conversion matrix between the previous frame image and the current frame image includes:

[0014] Determine a first camera pose according to a first IMU pose corresponding to the previous frame image and pre-calibrated camera extrinsic parameters, and determine a second camera pose according to a second IMU pose corresponding to the current frame image and the camera extrinsic parameters;

[0015] Determine a posture conversion matrix according to the first camera posture and the second camera posture;

[0016] The feature point conversion matrix is ​​determined according to the posture conversion matrix and the 3D coordinates of the first feature point corresponding to the previous frame image.

[0017] The beneficial effects of the above technical solution are as follows: since the IMU frame rate of the head-mounted display device is higher than the camera frame rate, and the IMU and camera have been calibrated when the device leaves the factory, the posture conversion matrix between the two frames of images can be determined based on the IMU posture and the camera external parameters, and then the feature point conversion matrix between the two frames of images can be determined in combination with the 3D coordinates of the extracted feature points, so as to use the feature point conversion matrix to track feature points on images of different scales, thereby improving the stability and robustness of the positioning algorithm.

[0018] Optionally, determining the feature point conversion matrix according to the posture conversion matrix and the 3D coordinates of the first feature point corresponding to the previous frame image includes:

[0019] Converting the first feature point corresponding to the previous frame image to the base layer of the scale pyramid, and decomposing along two axes of the image coordinate system to obtain a first decomposition result; the base layer is the level of the scale pyramid where the previous frame image is located;

[0020] Determine the 3D coordinates of the first feature point in the previous frame coordinate system according to the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after conversion to the base layer;

[0021] Convert the 3D coordinates of the first feature point in the current frame coordinate system to the image coordinate system, and decompose along two axes of the image coordinate system to obtain a second decomposition result;

[0022] The feature point conversion matrix is ​​determined according to the first decomposition result and the second decomposition result.

[0023] The beneficial effect of the above scheme is as follows: according to the first feature point extracted from each layer of the image contained in the scale pyramid corresponding to the posture transformation matrix and the previous frame image, the 3D coordinates of the first feature point in the current coordinate system are determined, and the movement of the feature point in the previous and next frames of images can be decomposed along the two axes of the image coordinate system, so as to determine the feature point transformation matrix using the decomposition results of the feature point in the previous and next frames of coordinate system. In this way, the feature points can be tracked on images of different scales according to the feature point transformation matrix to ensure that the extracted feature points are stable and have strong feature representation capabilities, thereby improving the stability and robustness of the overall positioning algorithm.

[0024] Optionally, determining the 3D coordinates of the first feature point in the previous frame coordinate system according to the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after conversion to the base layer includes:

[0025] According to the camera internal parameters, convert the 2D coordinates of the first feature point corresponding to the previous frame image into 3D coordinates on a normalized plane;

[0026] The acquired 3D coordinates of the first feature point are multiplied by corresponding components of the 3D coordinates after normalization of the plane to obtain the 3D coordinates of the first feature point in the previous frame coordinate system.

[0027] The beneficial effects of the above technical solution are as follows: considering the cumulative error of the visual odometer, the 3D coordinates of the first feature point obtained from the SLAM backend are not very accurate. Therefore, the 3D coordinates of the first feature point obtained are not used directly, but multiplied with the 3D coordinates on the normalized plane to determine the 3D coordinates of the first feature point in the coordinate system of the previous frame. Compared with directly using the obtained 3D coordinates, the accuracy of the 3D coordinates of the feature points can be improved, thereby improving the accuracy of feature point tracking and ensuring the accuracy of the positioning algorithm.

[0028] Optionally, determining, according to the feature point conversion matrix and the first feature point corresponding to the previous frame image, a second feature point matched by the first feature point in each layer image included in the scale pyramid of the current frame image includes:

[0029] Calculating a determinant value of the feature point conversion matrix, wherein the determinant value represents a degree of image scaling;

[0030] For the candidate layer image of the scale pyramid of the previous frame image, the following operations are performed, wherein the candidate layer image is any layer image in the scale pyramid of the previous frame image:

[0031] Determining a target level in the scale pyramid of the current frame image of the first feature point extracted from the candidate layer image according to the value of the determinant;

[0032] Feature point matching is performed on the first feature point extracted from the candidate layer image and the image corresponding to the target layer to obtain the second feature point in the image corresponding to the target layer.

[0033] The beneficial effect of the above technical solution is: since the calculation of the feature point conversion matrix is ​​calculated at the base level of the scale pyramid, for the first feature point in each layer of the scale pyramid of the previous frame image, it is possible to determine which layer of the scale pyramid of the current frame image it appears on based on the value of the determinant representing the degree of image scaling, thereby extracting the feature points on that layer of the image to achieve continuous and accurate tracking of the feature points.

[0034] Optionally, determining, according to the determinant value, a target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image includes:

[0035] The traversal starts from the base layer of the scale pyramid of the current frame image, and the following operations are performed in each traversal:

[0036] If the current level corresponding to the current determinant value is less than or equal to the number of levels of the scale pyramid, and the current determinant value does not exceed the scaling degree between two levels of the scale pyramid, the current level is taken as the target level; otherwise, the determinant value is reduced according to the scaling factor of the scale pyramid to traverse the next level of images.

[0037] The beneficial effect of the above technical solution is that when tracking feature points on images of different scales, the scale pyramid can be searched layer by layer by changing the determinant value of the feature point transformation matrix to determine the optimal layer where the feature points are located, so as to extract feature points on this layer of image, improve the accuracy of feature point matching on images of different scales, achieve continuous and accurate tracking of feature points, and thus improve the accuracy and robustness of the positioning algorithm.

[0038] Optionally, extracting line features from the current frame image includes:

[0039] Acquire a pre-generated mask image having the same pixel size as the current frame image, wherein the mask image includes a non-interested circular region with the center of the image as the origin and a preset number of pixels as the radius, wherein the preset number of pixels is less than the number of pixels of the short side of the current frame image;

[0040] Line features are extracted from the edge area of ​​the current frame image according to the mask image.

[0041] The beneficial effect of the above technical solution is: considering that the extraction of line features is time-consuming, a mask image is used to extract line features from the edge area of ​​each frame of the image, thereby ensuring the real-time performance of the positioning algorithm.

[0042] In a second aspect, an embodiment of the present application provides a head-mounted display device, including a processor, a memory, a multi-eye camera and an IMU, wherein the IMU, the multi-eye camera, the memory and the processor are connected via a bus;

[0043] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0044] Acquire an image sequence collected by a multi-camera, and construct a scale pyramid of a current frame image in the image sequence; the scale pyramid includes images of different scales generated after multiple downsampling of the current frame image;

[0045] Obtain the 3D coordinates of the first feature point corresponding to the previous frame image, and determine the feature point conversion matrix between the previous frame image and the current frame image in combination with the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image; wherein the first feature point is a feature point extracted from each layer of the image contained in the scale pyramid of the previous frame image, and the feature point includes a corner point and an edge point, the first IMU pose is calculated based on the data collected by the IMU corresponding to the previous frame image, and the second IMU pose is calculated based on the data collected by the IMU corresponding to the current frame image;

[0046] Determine, according to the feature point conversion matrix and the first feature point corresponding to the previous frame image, a second feature point matched by the first feature point in each layer image included in the scale pyramid of the current frame image;

[0047] Extracting line features from the current frame image;

[0048] Obtaining a 3D-2D point matching pair according to the line feature and the second feature point corresponding to the current frame image, and the line feature and the first feature point corresponding to the previous frame image;

[0049] The initial head pose corresponding to the current frame image is optimized according to the 3D-2D point matching pairs to obtain a target head pose; the initial head pose is determined according to the second IMU pose.

[0050] Optionally, the processor obtains the 3D coordinates of a first feature point corresponding to a previous frame image, combines the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image, and determines a feature point conversion matrix between the previous frame image and the current frame image. The specific operation is:

[0051] Determine a first camera pose according to a first IMU pose corresponding to the previous frame image and pre-calibrated camera extrinsic parameters, and determine a second camera pose according to a second IMU pose corresponding to the current frame image and the camera extrinsic parameters;

[0052] Determine a posture conversion matrix according to the first camera posture and the second camera posture;

[0053] The feature point conversion matrix is ​​determined according to the posture conversion matrix and the 3D coordinates of the first feature point corresponding to the previous frame image.

[0054] Optionally, the processor determines the feature point conversion matrix according to the posture conversion matrix and the 3D coordinates of the first feature point corresponding to the previous frame image, and the specific operation is:

[0055] Converting the first feature point corresponding to the previous frame image to the base layer of the scale pyramid, and decomposing along two axes of the image coordinate system to obtain a first decomposition result; the base layer is the level of the scale pyramid where the previous frame image is located;

[0056] Determine the 3D coordinates of the first feature point in the previous frame coordinate system according to the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after conversion to the base layer;

[0057] Determine the 3D coordinates of the first feature point in the current frame coordinate system according to the posture conversion matrix and the 3D coordinates of the first feature point in the previous frame coordinate system;

[0058] Convert the 3D coordinates of the first feature point in the current frame coordinate system to the image coordinate system, and decompose along two axes of the image coordinate system to obtain a second decomposition result;

[0059] The feature point conversion matrix is ​​determined according to the first decomposition result and the second decomposition result.

[0060] Optionally, the processor determines the 3D coordinates of the first feature point in the previous frame coordinate system according to the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after conversion to the base layer, and the specific operation is:

[0061] According to the camera internal parameters, convert the 2D coordinates of the first feature point corresponding to the previous frame image into 3D coordinates on a normalized plane;

[0062] The acquired 3D coordinates of the first feature point are multiplied by corresponding components of the 3D coordinates after normalization of the plane to obtain the 3D coordinates of the first feature point in the previous frame coordinate system.

[0063] Optionally, the processor determines, according to the feature point conversion matrix and the first feature point corresponding to the previous frame image, a second feature point matched by the first feature point in each layer image included in the scale pyramid of the current frame image, and the specific operation is:

[0064] Calculating a determinant value of the feature point conversion matrix, wherein the determinant value represents a degree of image scaling;

[0065] For the candidate layer image of the scale pyramid of the previous frame image, the following operations are performed, wherein the candidate layer image is any layer image in the scale pyramid of the previous frame image:

[0066] Determining a target level in the scale pyramid of the current frame image of the first feature point extracted from the candidate layer image according to the value of the determinant;

[0067] Feature point matching is performed on the first feature point extracted from the candidate layer image and the image corresponding to the target layer to obtain the second feature point in the image corresponding to the target layer.

[0068] Optionally, the processor determines, according to the value of the determinant, a target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image, and the specific operation is:

[0069] The traversal starts from the base layer of the scale pyramid of the current frame image, and the following operations are performed in each traversal:

[0070] If the current level corresponding to the current determinant value is less than or equal to the number of levels of the scale pyramid, and the current determinant value does not exceed the scaling degree between two levels of the scale pyramid, the current level is taken as the target level; otherwise, the determinant value is reduced according to the scaling factor of the scale pyramid to traverse the next level of images.

[0071] Optionally, the processor extracts line features from the current frame image, specifically by:

[0072] Acquire a pre-generated mask image having the same pixel size as the current frame image, wherein the mask image includes a non-interested circular region with the center of the image as the origin and a preset number of pixels as the radius, wherein the preset number of pixels is less than the number of pixels of the short side of the current frame image;

[0073] Line features are extracted from the edge area of ​​the current frame image according to the mask image.

[0074] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a head-mounted display device to execute any one of the positioning methods for head-mounted display devices provided in the embodiments of the present application.

[0075] The technical effects brought about by any one of the implementation methods in the second aspect to the third aspect can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0077] Figure 1 A schematic diagram of a head mounted display device provided in an embodiment of the present application;

[0078] Figure 2 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0079] Figure 3 A flow chart of a positioning method for a head mounted display device provided in an embodiment of the present application;

[0080] Figure 4 A schematic diagram of a scale pyramid constructed according to an embodiment of the present application;

[0081] Figure 5 This is a schematic diagram of fast corner feature extraction;

[0082] Figure 6 Schematic diagram of edge point feature extraction;

[0083] Figure 7 A flow chart of a method for determining a feature point transformation matrix;

[0084] Figure 8 A flow chart of a method for tracking feature points on images of different scales;

[0085] Fig. 9 Schematic diagram of the tracking effect of feature points on images of different scales;

[0086] Fig.10 is a schematic diagram of a mask image;

[0087] Fig.11 A schematic diagram of extracting corner points, edge points and line features provided in an embodiment of the present application;

[0088] Fig.12 The overall framework diagram of the SLAM positioning algorithm provided in the embodiment of the present application;

[0089] Fig.13 A structural diagram of a positioning device for a head mounted display device provided in an embodiment of the present application;

[0090] Fig.14 A structural diagram of a head-mounted display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0092] Based on the exemplary embodiments shown in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete technical solution separately.

[0093] In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0094] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0095] The terms used in the embodiments of the present application are explained below.

[0096] IMU pose: 6DoF pose calculated using data collected by the IMU, including translation and rotation.

[0097] The following is an overview of the design concept of the embodiments of the present application in conjunction with application scenarios.

[0098] VR / AR technology provides users with an immersive gaming experience. The translation and rotation of the user's head in three-dimensional space is an important part of the interaction in the VR / AR field. Accurately and stably determining the translation and rotation of the user's head can ensure the normal interaction of VR / AR applications. SLAM technology needs to extract enough and robust feature points to better find the correct correspondence. If enough and robust feature points cannot be extracted in scenes with weak textures, it will be difficult to ensure the accuracy and robustness of head tracking and positioning in scenes with weak textures, which will inevitably cause certain troubles in user experience, such as jitter and drift of the virtual scene, which will make it difficult to interact with the virtual scene accurately.

[0099] In view of this, the embodiment of the present application provides a positioning method for a head-mounted display device. For weak texture scenes, a scale pyramid is constructed for each frame of images collected by a multi-camera, and the features of the corner points and edge points of each layer of the scale pyramid are extracted to obtain scene features in images of different scales, and line features are extracted for each frame of the image. By fusing the corner point, edge point and line feature extraction algorithms, the number of points that can be tracked and matched in weak texture scenes is effectively increased, and more features can be added to the map to achieve dense mapping for robust head positioning. In order to further improve the accuracy of the matching points and enable the algorithm to run stably and robustly, after obtaining the 3D coordinates of the extracted feature points, the feature point conversion matrix is ​​determined according to the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image, and the feature points in each layer of the scale pyramid of the current frame image are tracked according to the feature point conversion matrix, thereby ensuring the continuous tracking of the same feature point in images of different scales, ensuring the stability and accuracy of feature point extraction, and then achieving accurate and robust head positioning.

[0100] See also Figure 1 , is a schematic diagram of a head-mounted display device provided in an embodiment of the present application. The head-mounted display device can be a VR device, an AR device, or an MR device. A multi-eye camera is arranged outside the head-mounted display device to collect images of the scene where the user is located, and an inertial measurement unit (IMU) is arranged inside the head-mounted display device to collect acceleration and angular velocity of the head. The acquisition frame rate of the IMU is higher than that of the multi-eye camera, and the IMU and the multi-eye camera have been calibrated before the device leaves the factory.

[0101] It should be noted that the embodiments of the present application do not impose any restrictive requirements on the structure of the head-mounted display device, and its multi-eye camera can be dual-eye or quad-eye.

[0102] See also Figure 2 , is a schematic diagram of an application scenario provided by an embodiment of the present application, in which a user wears a head-mounted display device for a 3D experience in an indoor scene with a smooth table. During the experience, the head-mounted display device implements a SLAM algorithm based on a visual-inertial odometry (VIO) composed of a multi-camera and an IMU to locate the head, and uses the positioning result to complete the interaction.

[0103] It should be noted that Figure 2 This is only an example of a scenario. The embodiment of the present application does not impose any restrictive requirements on the application scenario. It can be applicable to both indoor and outdoor scenarios.

[0104] It should be noted that the positioning algorithm of the embodiment of the present application only considers the feature matching between the frames of the multi-eye camera (that is, between the previous and next frames of each camera), and does not consider the feature matching within the frame of the multi-eye camera (that is, between the same frame of the multi-eye camera).

[0105] See also Figure 3 , is a flow chart of a positioning method for a head mounted display device provided in an embodiment of the present application. The process can be executed by the head mounted display device and mainly includes the following steps:

[0106] S301: Acquire an image sequence collected by a multi-camera, and construct a scale pyramid of a current frame image in the image sequence.

[0107] Specifically, for each image sequence captured by each camera, the following steps are performed: first, each frame of the image sequence is preprocessed into a grayscale image of a uniform data type (such as float32), and then the grayscale image is scaled to obtain a scale pyramid containing the original image and the transformed image. The grayscale image of the original image is used as the base image of the scale pyramid and is recorded as the 0th layer image.

[0108] Take the current frame image as an example, Figure 4 The figure shows a schematic diagram of the constructed scale pyramid. Assume that the current frame image is preprocessed into a grayscale image with a resolution of 640*480*1 (pixel). The grayscale image is downsampled 4 times, and the ratio of each downsampling is 1 / 2 of the image width and height, to obtain a 5-layer scale pyramid containing the original image. The higher the level of the pyramid, the smaller the image size and the lower the resolution.

[0109] S302: Obtain the 3D coordinates of the first feature point corresponding to the previous frame image, and determine the feature point conversion matrix between the previous frame image and the current frame image by combining the first IMU posture corresponding to the previous frame image and the second IMU posture corresponding to the current frame image.

[0110] In the embodiment of the present application, each time a scale pyramid of an image frame is constructed, feature points are extracted from each layer of the pyramid image, and the image level and coordinates of each extracted feature point are recorded.

[0111] Optionally, the feature point extraction of each layer of the image includes corner point feature extraction and edge point feature extraction.

[0112] In one example, the corner feature extraction algorithm uses a fast corner extraction algorithm. Since the fast corner extraction algorithm can detect places where local pixel grayscale changes significantly, it can ensure a certain extraction accuracy and is known for its fast extraction speed. Therefore, the fast corner extraction algorithm is selected to extract corner features.

[0113] like Figure 5The figure shows a schematic diagram of fast corner point extraction. For each pixel point in each layer of the scale pyramid image, the following is performed: for the pixel point p in the image, its gray value I is obtained. D , taking pixel p as the center, select 16 pixels on a circle with a radius of 3 pixels, and set the threshold T = 20%I p , if the brightness of 12 consecutive pixels among the 16 selected points is greater than I p +T or less than I D -T, the pixel point p is considered to be a feature corner point.

[0114] In one example, edge point features may be extracted based on the gradient information of the grayscale image. Specifically, for each layer of the scale pyramid image, the following operations are performed:

[0115] (1) Calculate the first-order derivative of the image Img and perform Gaussian blur on the image.

[0116] (2) Use the edge detection operator to calculate the first-order derivatives of the image in the x and y directions to obtain the gradient of each pixel in the x and y directions.

[0117] Among them, edge detection operators include but are not limited to Scharr operator, Sobel operator and Canny operator. Taking Scharr operator as an example, its convolution kernel is expressed as:

[0118]

[0119] At this time, the gradient of each pixel (x1, y1) is:

[0120] grad x1 =(-3*Img(x1-1,y1-1)+3*Img(x1+1,y1-1)-10*Img(x1-1,y1)+10*Img(x1+1,y1)-3*Img(x1-1,y1+1)+3*Img(x1+1,y1+1)) / 3;

[0121] grad y1 =(-3*Img(x1-1, y1-1)-10*Img(x1, y1-1)+3*Img(x1+1, y1-1)-3*Img(x1-1, y1+1)+10*Img(x1, y1+1)+3*Img(x1+1, y1+1)) / 3.

[0122] (3) According to the gradient of each pixel in the x-direction and the y-direction, the angle and amplitude of the corresponding pixel are determined. The calculation formula is as follows:

[0123]

[0124] in, The unit is radian, an g The unit angle of le1. The replication of each pixel can be recorded using a table with the same size as the resolution of the image Img.

[0125] (4) Filter out edge points based on the amplitude of each pixel.

[0126] Specifically, pixel points around the pixel point are selected and amplitude comparison is performed. If the amplitude of the pixel point is the largest and the amplitude of the pixel point is greater than a preset change threshold, the pixel point is determined to be an edge point.

[0127] like Figure 6 The figure shows a schematic diagram of the 9-point comparison method, in which the amplitude of the pixel point and the surrounding pixels are judged respectively.

[0128] In one example, in order to facilitate the management and use of feature points, after extracting the corner points and edge points in each layer of the scale pyramid image, the coordinates of each feature point can be recorded through the grid image. Since both corner points and edge points are feature points, the coordinate recording method does not need to distinguish between corner points and edge points, and the recording methods of the two are the same.

[0129] Specifically, a new image with the same size as the feature extraction image and with all pixel values ​​0 is created (for example, for a scale image with a resolution of 640*480, the resolution of the new image is also 640*480), and the values ​​of the pixels at the same position are changed to 1 according to the coordinates of the extracted feature points to obtain a grid image. The grid image, as an intermediate result, can be used in subsequent feature matching and tracking processes. Compared with directly recording the coordinates of the feature points, it is easier to perform batch operations and feature recovery.

[0130] Based on the above-mentioned method of extracting corner point features and edge point features, feature points are extracted for each frame of image, wherein the feature points extracted from each layer of images included in the scale pyramid of the previous frame of image are recorded as first feature points, and the feature points extracted from each layer of images included in the scale pyramid of the current frame of image are recorded as second feature points.

[0131] Each feature point can be identified by a unique ID, and the coordinates and image level of the corresponding feature point can be obtained through the ID.

[0132] Since the identification of the same feature point in the images collected by the multi-camera at different times is the same, and after each feature point extraction is completed, its 3D coordinates can be calculated by the SLAM backend. Therefore, for the first feature point corresponding to the previous frame of the image, its 3D coordinates can be obtained from the SLAM backend based on the ID of the first feature point.

[0133] Considering that when the user moves, the scale of the feature will change with the movement (for example, a box appears in front of the user, and the user moves back and forth towards the box. When the user is close to the box, the box tends to be an enlarged square feature. Conversely, when the user is away from the box, the box tends to be a reduced point feature). Therefore, in order to ensure the accuracy of feature point extraction, the tracking of feature points in the previous and next frame images in the embodiment of the present application is continuous tracking under different scale images of the pyramid.

[0134] Since the IMU and multi-eye camera in the head-mounted display device are calibrated before the device leaves the factory, the camera intrinsic parameters and extrinsic parameters can be obtained through the calibration file, and the acquisition frame rate of the IMU is usually greater than the acquisition frame rate of the multi-eye camera. Therefore, the posture determined by the IMU data is combined with the intrinsic and extrinsic parameters of the camera to determine the posture conversion matrix between the previous and next frames, and then the feature point conversion matrix is ​​determined according to the extracted feature points, so as to track the feature points on the different scale images of the previous and next frames, thereby improving the stability and robustness of the positioning algorithm.

[0135] In one example, the feature point transformation matrix is ​​determined as follows: Figure 7 As shown, it mainly includes the following steps:

[0136] S3021: Determine a first camera pose according to a first IMU pose corresponding to a previous frame image and pre-calibrated camera extrinsic parameters, and determine a second camera pose according to a second IMU pose corresponding to a current frame image and camera extrinsic parameters.

[0137] Assuming that the time corresponding to the previous frame image is t0, and the time corresponding to the current frame image is t1, the first IMU pose calculated based on the IMU data at time t0 is The second IMU pose calculated based on the IMU data at time t1 is The pre-calibrated IMU to camera extrinsic parameters are Then the first camera pose at time t0 is:

[0138]

[0139] The position of the second camera at time t1 is:

[0140]

[0141] S3022: Determine a posture conversion matrix according to the first camera posture and the second camera posture.

[0142] Among them, the pose conversion matrix between the frames before and after t0 and t1 is expressed as:

[0143]

[0144] S3023: Determine a feature point conversion matrix according to the posture conversion matrix and the 3D coordinates of the first feature point corresponding to the previous frame image.

[0145] For each layer of the scale pyramid of the previous frame image, the first feature point corresponding to the previous frame image can be obtained according to the feature extraction method of the corner points and edge points mentioned above. Through the SLAM backend, the 3D coordinates of each first feature point in the world coordinate system can be obtained, and then combined with the pose transformation matrix, the feature point transformation matrix between the previous and next frames can be calculated.

[0146] It should be noted that since corner points and edge points are both point features, when calculating the feature point transformation matrix, corner points and edge points are no longer distinguished and are all treated as feature points.

[0147] Specifically, the calculation process of the feature point transformation matrix is ​​as follows:

[0148] S3023_1: convert the first feature point corresponding to the previous frame image to the base layer of the scale pyramid, and decompose it along two axes of the image coordinate system to obtain a first decomposition result.

[0149] Among them, the base layer is the 0th layer of the scale pyramid. The image of this layer retains the resolution of the original image, and the images of other layers are the results of downsampling the image of this layer. Therefore, the coordinate transformation of feature points between images of different scales can be performed according to the scaling method between images when constructing the scale pyramid, and the image level of the feature points remains unchanged.

[0150] Taking the example of downsampling the scale pyramid at a ratio of 1 / 2 of the image width and height each time, assuming that the coordinates of the feature point extracted from the image of the nth layer (n=1, 2, 3..) of the scale pyramid are (x, y), then the coordinates of the feature point converted to the 0th layer image are (2n*x, 2n*y).

[0151] When the user's head moves, the feature points will also move with it. The movement vector of each feature point can be decomposed along the U axis and V axis of the image coordinate system. Assume that the first feature point extracted from each layer of the scale pyramid of the previous frame image at time t0 moves p along the U axis and V axis respectively. du_t0 (du, 0) and p dv_t0 (0, dv).

[0152] S30232: Determine the 3D coordinates of the first feature point in the previous frame coordinate system according to the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after conversion to the base layer.

[0153] The process of determining the 3D coordinates of the first feature point in the previous frame coordinate system is as follows:

[0154] First, according to the intrinsic parameters of the camera, the 2D coordinates of the first feature point corresponding to the previous frame image are converted into 3D coordinates on the normalized plane.

[0155] Assume that the time corresponding to the previous frame image is t0, and take the first feature point p corresponding to time t0 t0 For example, assume that its 2D coordinate in the image is p t0 (u, v), then p t0 (u, v) is converted to 3D coordinates on the normalized plane: in, f x 、f y 、c x and c y is the pre-calibrated camera intrinsic parameter, f x and f y is the camera equivalent focal length, c x and c y The optical center of the camera.

[0156] Then, the acquired 3D coordinates of the first feature point are multiplied by corresponding components of the 3D coordinates on the normalized plane to obtain the 3D coordinates of the first feature point in the previous frame coordinate system.

[0157] Assume that the first feature point p obtained from the SLAM backend t0 The 3D coordinates of the first feature point p are P0(x0, y0, z0). t0 3D coordinates on the normalized plane The 3D coordinate converted to the previous frame coordinate system is P t0 (x t0 ,y t0 , z t0 ), where x t0 =x*z0,y t0 =y*z0,z t0 =z0.

[0158] In an embodiment of the present application, taking into account the cumulative error of the visual odometer, the 3D coordinates of the first feature point obtained from the SLAM backend are not very accurate. Therefore, the 3D coordinates of the first feature point obtained are not used directly. Instead, the 3D coordinates of the first feature point are multiplied with the 2D coordinates on the image on the normalized plane to determine the 3D coordinates of the first feature point in the coordinate system of the previous frame. Compared with directly using the acquired 3D coordinates, the accuracy of the 3D coordinates of the feature point can be improved, thereby improving the accuracy of feature point tracking and ensuring the accuracy of the positioning algorithm.

[0159] S30233: Determine the 3D coordinates of the first feature point in the current frame coordinate system according to the posture conversion matrix and the 3D coordinates of the first feature point in the previous frame coordinate system.

[0160] Using the pose conversion matrix, the 3D coordinates of the first feature point in the previous frame coordinate system are converted to the current frame coordinate system to obtain the 3D coordinates P of the first feature point in the current frame coordinate system. t1 (x t1 ,y t1 , z t1 ), the formula is:

[0161]

[0162] S3023_4: Convert the 3D coordinates of the first feature point in the current frame coordinate system to the image coordinate system, and decompose along two axes of the image coordinate system to obtain a second decomposition result.

[0163] According to the pre-calibrated camera internal parameters, the 3D coordinates P of the first feature point in the current frame coordinate system can be t1 (x t1 ,y t1 , z t1 ) is converted to the image coordinate system at time t1, and the 2D coordinate p of the first feature point at time t1 is obtained t1 (u, v), the formula is:

[0164]

[0165] Similarly, by decomposing pt1(u, v) along the U axis and V axis of the image coordinate system, we can obtain the movement components of the first feature point at time t1 as p du_t1 and p dv_t1 .

[0166] S30235: Determine a feature point conversion matrix according to the first decomposition result and the second decomposition result.

[0167] Assume that the conversion matrix of the first feature point extracted at time t0 to the feature point at time t1 is expressed as Then p at time t0 du_t0 The formula for conversion to time t1 is expressed as Right now:

[0168]

[0169] Similarly, let p at time t0 dv_t0 Switch to time t1 and insert p dv_t1 It can be solved

[0170] In an embodiment of the present application, the 3D coordinates of the first feature point in the current coordinate system are determined based on the posture transformation matrix and the first feature point in each layer of the scale pyramid of the previous frame image, and the movement of the feature point in the previous and next frames of images can be decomposed along the two axes of the image coordinate system, thereby using the decomposition results of the feature point in the previous and next frames of coordinate system to determine the feature point transformation matrix. In this way, the feature points can be tracked on images of different scales according to the feature point transformation matrix, ensuring that the same feature point can be continuously and stably tracked in images of different scales, extending the "lifespan" of the extracted feature points, ensuring that the extracted feature points are stable and have strong feature representation capabilities, thereby improving the stability and robustness of the overall positioning algorithm.

[0171] S303: Determine, according to the feature point conversion matrix and the first feature point corresponding to the previous frame image, the second feature point that matches the first feature point in each layer of the image included in the scale pyramid of the current frame image.

[0172] Since the solution of the feature point conversion matrix is ​​converted to the 0th layer of the scale pyramid, in order to realize the tracking of feature points on images of different scales, it is necessary to determine at which layer the first feature point extracted at time t0 (i.e., the scale pyramid of the previous frame image) appears at time t1 (i.e., the scale pyramid of the current frame image). The determinant of the feature point conversion matrix can represent the degree of scaling between images of different layers of the scale pyramid. Therefore, the best tracking level of the first feature point extracted from the scale pyramid of the previous frame image in the scale pyramid of the current frame image can be determined according to the determinant of the feature point conversion matrix. In this way, for each layer of the scale pyramid of the previous frame image, the image of the layer, the first feature point extracted from the layer image, and the image of the corresponding level of the first feature point in the layer image in the scale pyramid of the current frame image are used as inputs, and the optical flow matching method is used to accurately track the feature points at the level of the scale pyramid of the current frame image, thereby realizing the tracking of feature points on images of different scales.

[0173] For details, see the feature point tracking process. Figure 8 , mainly includes the following steps:

[0174] S3031: Calculate the determinant value of the feature point conversion matrix.

[0175] When the user's head moves, the scale of the feature will also change. To determine the scale of the feature after the change, the determinant of the feature point transformation matrix can be calculated.

[0176] Since the calculation of the feature point conversion matrix is ​​performed by converting to the 0th layer of the scale pyramid, the scale represented by the row and column values ​​at this time is the scale of the original image, that is, before scaling.

[0177] S3032: Determine, according to the determinant value, a target level of the scale pyramid of the current frame image for the first feature point extracted from the candidate layer image; wherein the candidate layer image is any layer image in the scale pyramid of the previous frame image.

[0178] For the first feature points extracted from the candidate layer image, in order to achieve tracking on images of different scales, it is necessary to determine the target level at which each first feature point appears in the scale pyramid of the current frame image. Specifically, the traversal can be started from the base layer (i.e., layer 0) of the scale pyramid of the current frame image to determine the target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image, wherein each traversal performs the following operations:

[0179] S3032_1: Determine whether the current level corresponding to the current determinant value in the scale pyramid of the current frame image is less than or equal to the number of scale pyramid levels. If so, execute S30322; otherwise, execute S30325.

[0180] During the traversal process, it is necessary to ensure that it is performed inside the scale pyramid of the current frame image, so it is necessary to ensure that the current level is less than or equal to the number of levels of the scale pyramid.

[0181] S30322: Determine whether the current determinant value does not exceed the scaling degree between two layers of the scale pyramid. If so, execute S30323; otherwise, execute S30324.

[0182] Assuming that the scaling degree between two adjacent layers in the scale pyramid is 2, if the current determinant value exceeds 3, it is equivalent to skip-layer search, so it is necessary to ensure that the current determinant value does not exceed the scaling degree between the two layers of the scale pyramid.

[0183] S30323: Set the current level as the target level and end the traversal.

[0184] S30324: Reduce the determinant value according to the scaling factor of the scale pyramid to traverse the next level of images.

[0185] When the determinant value exceeds the scaling degree between two levels of the scale pyramid and the current level does not exceed the number of levels of the scale pyramid, the optimal target level is determined by gradually reducing D. Each time the determinant value is reduced, the traversed image level increases.

[0186] Assume that when constructing the scale pyramid, the image is reduced by 1 / 2 in the U and V directions during each downsampling process, so the determinant value of the feature point transformation matrix in each layer of the image is reduced by (1 / 2)*(1 / 2) each time, that is, the determinant value is reduced by 0.25 each time.

[0187] S30325: Use the base level of the scale pyramid as the target level.

[0188] After the traversal is completed, when no determinant value is found that does not exceed the scaling degree between the two layers of the scale pyramid and is less than or equal to the level of the scale pyramid, the base layer (ie, the 0th layer) is used as the target level.

[0189] In the embodiments of the present application, since the feature point conversion matrix is ​​calculated at the base level of the scale pyramid, when tracking feature points on images of different scales, the scale pyramid can be searched layer by layer by changing the determinant value of the feature point conversion matrix to determine the optimal level at which the feature points are located, thereby extracting feature points on the image of that level, improving the accuracy of matching feature points on images of different scales, achieving continuous and accurate tracking of feature points, and thereby improving the accuracy and robustness of the positioning algorithm.

[0190] S3033: Perform feature point matching on the first feature point extracted from the candidate layer image and the image corresponding to the target layer to obtain the second feature point in the image corresponding to the target layer.

[0191] In one example, the feature extraction method of corner points and edge points described above is used to extract second feature points in the image corresponding to the target level in the scale pyramid of the current frame image, and the descriptors of the second feature points extracted in the image corresponding to the target level and the descriptors of the first feature points extracted in the candidate layer image are calculated, and the feature points in the currently extracted second feature points that match the first feature points are determined based on the descriptors.

[0192] like Fig. 9 As shown, it is a schematic diagram of the tracking effect of feature points on images of different scales. The first feature point extracted from the 0th layer image of the scale pyramid of the previous frame image is matched with the second feature point extracted from the 2nd layer image of the scale pyramid of the current frame image.

[0193] In an embodiment of the present application, the determinant value of the feature point transformation matrix is ​​used to determine the level of the scale pyramid, thereby achieving tracking of feature points on images of different scales, ensuring that feature point feature changes caused by movement changes between previous and subsequent frames are not easily lost, and improving the robustness of feature point matching between frames.

[0194] S304: extracting line features from the current frame image.

[0195] In order to add more features to improve the accuracy of the positioning algorithm, in addition to matching feature points on images of different scales, line feature extraction is also performed. Different from feature point extraction, considering that line feature extraction is time-consuming, this application extracts line features for each frame of image captured by the multi-camera.

[0196] In one example, considering that the handover process has a high requirement on the response speed of the head-mounted display device, in order to ensure the real-time performance of the positioning algorithm, line features may be extracted only from the edge area of ​​each frame of the image.

[0197] Taking the line feature extraction process of the current frame image as an example, a pre-generated mask image with the same pixel size as the current frame image is obtained, and line feature extraction is performed on the edge area of ​​the current frame image according to the mask image. The mask image includes a circular area with the center of the image as the origin and a preset number of pixels as the radius. The circular area is used as an area of ​​no interest. The preset number of pixels is less than the number of pixels of the short side of the current frame image so that the edge area is retained, and line feature extraction is performed only on the edge area.

[0198] like Fig.10 As shown, it is a schematic diagram of the mask image. Assuming that the resolution of the current frame image is 640*480 (pixels), the size of the mask image is also 640*480 (pixels), and the radius of the circular area in the center is 180 pixels.

[0199] In specific implementation, the line feature extraction process includes:

[0200] First, the gradient of each pixel in the current frame image is calculated.

[0201] In order to record the changes in light and dark and obtain the position of the edge of the line segment, the gradient of each pixel can be calculated. The calculation formula is as follows:

[0202]

[0203] Then, according to the gradient of each pixel in the current frame image, the angle and amplitude of the corresponding pixel are calculated. The calculation formula is as follows:

[0204]

[0205] It can be seen from Formula 11 and Formula 1 that the gradient direction of the line feature is perpendicular to the gradient direction of the edge point.

[0206] Secondly, at least one straight line area is determined according to the angle of each pixel point.

[0207] Since a line is made up of points and each pixel has a direction (angle), by calculating the difference between every two pixels, at least one straight line area can be determined.

[0208] Specifically, a pixel point is selected from each pixel point in the edge area, and the angle of the pixel point is used as the initial angle of a straight line area. The angle difference between the pixel point and the surrounding pixels is calculated. If the angle difference between the two is less than a preset empirical value (such as 22°), the corresponding surrounding pixels can be included in the straight line area. Each time a new pixel point is included, the angle of the straight line area is updated once, and the update method is as follows:

[0209]

[0210] Among them, i is the number of included pixels.

[0211] In one example, when the endpoint coordinates of a straight line region are located in a circular region of a mask image, the straight line region is discarded, thereby achieving line feature extraction only in the edge region of the current frame image. By extracting line features only in the edge region of the image, the positioning algorithm can maintain real-time and stability, and the addition of line features also makes the positioning algorithm more robust.

[0212] Finally, rectangle calculation is performed based on at least one straight line area to obtain line features.

[0213] After obtaining at least one straight line area, each straight line area is regarded as a rectangular rigid body, and the amplitude of the pixel points in the straight line area is regarded as the mass of the straight line area. Therefore, the mass center of each straight line area corresponds to the center of a rectangle, and the width and length of the rectangle are the minimum values ​​that allow the rectangle to cover the entire straight line area. Among them, the direction of the rectangle is the updated angle line , the center of the rectangle is (cen x ,cen y ), the formula is as follows:

[0214]

[0215]

[0216] Through the calculation of the above rectangle, line features can be extracted.

[0217] like Fig.11 As shown, it is a schematic diagram of the extraction of corner points, edge points and line features provided in an embodiment of the present application. By extracting corner point features and edge point features on images of different scales, and extracting line features in the acquired image, it is possible to make up for the problem of insufficient feature point extraction in weak texture scenes (such as a small room with only a few objects placed in it, surrounded by white walls, and other areas with blurred spatial features), thereby ensuring the robustness of positioning and mapping in weak texture scenes.

[0218] S305: Obtain 3D-2D point matching pairs according to the line features and the second feature points corresponding to the current frame image, and the line features and the first feature points corresponding to the previous frame image.

[0219] The second feature point tracked in the scale pyramid of the current frame image and the first feature point matched in the scale pyramid of the previous frame image are the same 3D point in the three-dimensional space. After the first feature point is extracted, the optimized 3D coordinates can be obtained through the SLAM backend. These 3D coordinates and the 2D coordinates of the second feature point can form a 3D-2D point matching pair, and the line feature can be regarded as composed of the features of each pixel point. Therefore, the line features in the current frame image and the line features in the previous frame image can be split into point features, and then combined with the 3D coordinates of the points contained in the line features obtained from the SLAM backend, they can also form a 3D-2D point matching pair.

[0220] S306: Optimize the initial head pose corresponding to the current frame image according to the 3D-2D point matching to obtain the target head pose.

[0221] The initial head pose of the current frame image collected at time t1 can be determined by the second IMU pose calculated from the data collected by the IMU at time t1 and the pre-calibrated camera external parameters. The SALM backend uses the 3D-2D point matching pairs determined by S305 to optimize the pose and obtain the target head pose.

[0222] See also Fig.12 , which is the overall framework diagram of the SLAM positioning algorithm provided in the embodiment of the present application, mainly includes data acquisition, front-end estimation, back-end optimization, loop detection and map construction. The data acquisition part mainly uses the camera for image acquisition; the front-end estimation part is used for feature extraction and providing initial values ​​of posture. In the feature extraction process, a scale pyramid is constructed for each frame of the image, and corner point features and edge point features are extracted in the scale pyramid, and the extracted feature points are tracked on different scale images of the previous and next frames, and line features are extracted in the 0th layer of the scale pyramid, and 3D-2D matching point pairs are generated based on the extracted line features and feature points; the back-end optimization part mainly uses the information of 3D-2D matching point pairs and loop detection for posture optimization, and the map construction part mainly uses the optimized posture trajectory to construct a map.

[0223] In the embodiment of the present application, in order to extract enough feature points for the head-mounted display device in the weak texture scene, a scale pyramid is constructed for each frame of the image collected by the multi-eye camera, and the corner point and edge point features are extracted for each layer of the scale pyramid, and the line feature extraction is performed for each frame of the image. By fusing the corner point, edge point and line feature extraction algorithms, the number of feature points in the weak texture scene is effectively increased. On the other hand, in order to further improve the accuracy of feature point matching and enable the algorithm to run stably and robustly, the feature point conversion matrix is ​​calculated according to the IMU pose corresponding to the previous and next frame images and the 3D coordinates of the extracted feature points, and the feature point conversion matrix is ​​used to track the feature points of the scale pyramid of the previous and next frame images, thereby ensuring the continuous and stable tracking of the same feature point under different scale images, and ensuring the stability and accuracy of feature point extraction. In this way, when the head pose is optimized according to the 3D-2D point matching pairs composed of the extracted feature points and line features, the stability and robustness of the SLAM algorithm can be effectively improved, thereby ensuring the accuracy and robustness of the positioning results, so that the wearer of the head-mounted display device can obtain a good interactive experience.

[0224] Based on the same technical concept, the embodiment of the present application provides a positioning device for a head mounted display device, the structure of which is as follows: Fig.13 As shown, it includes a pyramid construction module 1301, a conversion matrix calculation module 1302, a feature extraction module 1303 and a posture optimization module 1304, wherein:

[0225] The pyramid construction module 1301 is used to obtain an image sequence collected by a multi-camera and construct a scale pyramid of a current frame image in the image sequence; the scale pyramid includes images of different scales generated after multiple downsampling of the current frame image;

[0226] The conversion matrix calculation module 1302 is used to obtain the 3D coordinates of the first feature point corresponding to the previous frame image, and determine the feature point conversion matrix between the previous frame image and the current frame image in combination with the first IMU posture corresponding to the previous frame image and the second IMU posture corresponding to the current frame image; wherein the first feature point is a feature point extracted from each layer of the image contained in the scale pyramid of the previous frame image, and the feature point includes a corner point and an edge point;

[0227] The feature extraction module 1303 is used to determine, based on the feature point conversion matrix and the first feature point corresponding to the previous frame image, the second feature point matched by the first feature point in each layer of the image included in the scale pyramid of the current frame image; and extract the line feature from the current frame image, and obtain a 3D-2D point matching pair based on the line feature and the second feature point corresponding to the current frame image, and the line feature and the first feature point corresponding to the previous frame image;

[0228] The posture optimization module 1304 is used to optimize the initial head posture corresponding to the current frame image according to the 3D-2D point matching to obtain the target head posture; the initial head posture is determined according to the second IMU posture.

[0229] Optionally, the conversion matrix calculation module 1302 is specifically used for:

[0230] Determine a first camera pose according to a first IMU pose corresponding to a previous frame image and pre-calibrated camera extrinsic parameters, and determine a second camera pose according to a second IMU pose corresponding to a current frame image and camera extrinsic parameters;

[0231] Determine a posture conversion matrix according to the first camera posture and the second camera posture;

[0232] Determine the feature point conversion matrix according to the posture conversion matrix and the 3D coordinates of the first feature point corresponding to the previous frame image.

[0233] Optionally, the conversion matrix calculation module 1302 is specifically used for:

[0234] The first feature point corresponding to the previous frame image is converted to the base layer of the scale pyramid, and decomposed along the two axes of the image coordinate system to obtain a first decomposition result; the base layer is the level of the scale pyramid where the previous frame image is located;

[0235] Determine the 3D coordinates of the first feature point in the previous frame coordinate system according to the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after conversion to the base layer;

[0236] Determine the 3D coordinates of the first feature point in the current frame coordinate system according to the posture conversion matrix and the 3D coordinates of the first feature point in the previous frame coordinate system;

[0237] The 3D coordinates of the first feature point in the current frame coordinate system are converted to the image coordinate system, and decomposed along two axes of the image coordinate system to obtain a second decomposition result;

[0238] A feature point conversion matrix is ​​determined according to the first decomposition result and the second decomposition result.

[0239] Optionally, the conversion matrix calculation module 1302 is specifically used for:

[0240] According to the camera internal parameters, the 2D coordinates of the first feature point corresponding to the previous frame image are converted into 3D coordinates on the normalized plane;

[0241] The acquired 3D coordinates of the first feature point are multiplied by the corresponding components of the 3D coordinates after the normalization plane to obtain the 3D coordinates of the first feature point in the previous frame coordinate system.

[0242] Optionally, the feature extraction module 1303 is specifically used for:

[0243] Calculate the determinant value of the feature point conversion matrix, the determinant value represents the image scaling degree;

[0244] For the candidate layer image of the scale pyramid of the previous frame image, the following operations are performed, where the candidate layer image is any layer image in the scale pyramid of the previous frame image:

[0245] According to the determinant value, determine the target level of the first feature point extracted from the candidate layer image in the scale pyramid of the current frame image;

[0246] Feature point matching is performed on the first feature point extracted from the candidate layer image and the image corresponding to the target layer to obtain the second feature point in the image corresponding to the target layer.

[0247] Optionally, the feature extraction module 1303 is specifically used for:

[0248] Starting from the base layer of the scale pyramid of the current frame image, each traversal performs the following operations:

[0249] If the current level corresponding to the current determinant value is less than or equal to the number of levels of the scale pyramid, and the current determinant value does not exceed the scaling degree between the two levels of the scale pyramid, the current level is taken as the target level. Otherwise, the determinant value is reduced according to the scaling factor of the scale pyramid to traverse the next level of images.

[0250] For the convenience of description, the head mounted display device can be divided into modules (or units) according to their functions and described separately. Of course, when implementing the present application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0251] Based on the same technical concept, an embodiment of the present application provides a head-mounted display device that can implement the steps of the positioning method of the above-mentioned head-mounted display device and achieve the same technical effect.

[0252] See also Fig.14 , the head mounted display device includes a processor 1401, a memory 1402, a multi-eye camera 1403 and an IMU 1404, and the IMU 1404, the multi-eye camera 1403, the memory 1402 and the processor 1401 are connected via a bus 1405;

[0253] IMU 1404 is used to provide data for calculating IMU posture;

[0254] The multi-eye camera 1403 is used to collect images;

[0255] The memory 1002 stores a computer program, and the processor 1001 executes the steps of the positioning method of the head-mounted display device provided in the embodiment of the present application according to the computer program.

[0256] In the embodiment of the present application, the memory 1402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc. The memory 1402 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 1402 may also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1402 may be any other medium that can be used to carry or store a desired computer program in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1402 may be a combination of the above memories.

[0257] The processor 1401 may include one or more central processing units (CPU), GPU or digital processing unit, etc. The processor 1401 is used to implement the steps of any of the above-mentioned positioning methods for head mounted display devices when calling the computer program stored in the memory 1402 .

[0258] It should be noted that Fig.14 It is only an example to provide the necessary hardware for the head mounted display device to execute the steps of the positioning method for the head mounted display device provided in the embodiment of the present application. If not shown, the head mounted display device may also include conventional hardware such as a display screen, a speaker, a pickup, a communication interface, a power supply, a handle, and buttons.

[0259] In the embodiment of the present application, the specific connection medium between the IMU 1404, the multi-eye camera 1403, the memory 1402 and the processor 1401 is not limited. In the embodiment of the present application, the bus 1405 between the IMU 1404, the multi-eye camera 1403, the memory 1402 and the processor 1401 is Fig.14 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1405 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.14 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0260] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0261] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions. When these instructions are executed, the steps of the positioning method of any head-mounted display device in the aforementioned embodiments can be completed.

[0262] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute the steps of the positioning method for a head mounted display device in any of the aforementioned embodiments.

[0263] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0264] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0265] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0266] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0267] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for positioning a head mounted display device, characterized in that: include: Acquire an image sequence captured by a multi-camera, and construct a scale pyramid of a current frame image in the image sequence; The scale pyramid includes images of different scales generated by multiple downsampling of the current frame image; Obtain the 3D coordinates of the first feature point corresponding to the previous frame image, and determine the feature point conversion matrix between the previous frame image and the current frame image in combination with the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image; wherein the first feature point is a feature point extracted from each layer of the image included in the scale pyramid of the previous frame image, and the feature point includes a corner point and an edge point; Determine, according to the feature point conversion matrix and the first feature point corresponding to the previous frame image, a second feature point matched by the first feature point in each layer image included in the scale pyramid of the current frame image; Extracting line features from the current frame image; Obtaining a 3D-2D point matching pair according to the line feature and the second feature point corresponding to the current frame image, and the line feature and the first feature point corresponding to the previous frame image; The initial head pose corresponding to the current frame image is optimized according to the 3D-2D point matching pairs to obtain a target head pose; the initial head pose is determined according to the second IMU pose.

2. The method according to claim 1, characterized in that The step of obtaining the 3D coordinates of the first feature point corresponding to the previous frame image, combining the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image, and determining the feature point conversion matrix between the previous frame image and the current frame image includes: Determine a first camera pose according to a first IMU pose corresponding to the previous frame image and pre-calibrated camera extrinsic parameters, and determine a second camera pose according to a second IMU pose corresponding to the current frame image and the camera extrinsic parameters; Determine a posture conversion matrix according to the first camera posture and the second camera posture; The feature point conversion matrix is ​​determined according to the posture conversion matrix and the 3D coordinates of the first feature point corresponding to the previous frame image.

3. The method according to claim 2, characterized in that The step of determining the feature point conversion matrix according to the posture conversion matrix and the 3D coordinates of the first feature point corresponding to the previous frame image includes: Converting the first feature point corresponding to the previous frame image to the base layer of the scale pyramid, and decomposing along two axes of the image coordinate system to obtain a first decomposition result; the base layer is the level of the scale pyramid where the previous frame image is located; Determine the 3D coordinates of the first feature point in the previous frame coordinate system according to the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after conversion to the base layer; Determine the 3D coordinates of the first feature point in the current frame coordinate system according to the posture conversion matrix and the 3D coordinates of the first feature point in the previous frame coordinate system; Convert the 3D coordinates of the first feature point in the current frame coordinate system to the image coordinate system, and decompose along two axes of the image coordinate system to obtain a second decomposition result; The feature point conversion matrix is ​​determined according to the first decomposition result and the second decomposition result.

4. The method according to claim 3, characterized in that Determining the 3D coordinates of the first feature point in the previous frame coordinate system according to the 3D coordinates of the first feature point and the 2D coordinates of the first feature point after conversion to the base layer includes: According to the camera internal parameters, convert the 2D coordinates of the first feature point corresponding to the previous frame image into 3D coordinates on a normalized plane; The acquired 3D coordinates of the first feature point are multiplied by corresponding components of the 3D coordinates after normalization of the plane to obtain the 3D coordinates of the first feature point in the previous frame coordinate system.

5. The method according to claim 1, characterized in that The determining, according to the feature point conversion matrix and the first feature point corresponding to the previous frame image, a second feature point matched by the first feature point in each layer image included in the scale pyramid of the current frame image comprises: Calculating a determinant value of the feature point conversion matrix, wherein the determinant value represents a degree of image scaling; For the candidate layer image of the scale pyramid of the previous frame image, the following operations are performed, wherein the candidate layer image is any layer image in the scale pyramid of the previous frame image: Determining a target level in the scale pyramid of the current frame image of the first feature point extracted from the candidate layer image according to the value of the determinant; Feature point matching is performed on the first feature point extracted from the candidate layer image and the image corresponding to the target layer to obtain the second feature point in the image corresponding to the target layer.

6. The method according to claim 5, characterized in that The step of determining, according to the determinant value, a target level in the scale pyramid of the current frame image of the first feature point extracted from the candidate layer image comprises: The traversal starts from the base layer of the scale pyramid of the current frame image, and the following operations are performed in each traversal: If the current level corresponding to the current determinant value is less than or equal to the number of levels of the scale pyramid, and the current determinant value does not exceed the scaling degree between two levels of the scale pyramid, the current level is taken as the target level; otherwise, the determinant value is reduced according to the scaling factor of the scale pyramid to traverse the next level of images.

7. The method according to claim 1, characterized in that The extracting line features from the current frame image includes: Acquire a pre-generated mask image having the same pixel size as the current frame image, wherein the mask image includes a non-interested circular region with the center of the image as the origin and a preset number of pixels as the radius, wherein the preset number of pixels is less than the number of pixels of the short side of the current frame image; Line features are extracted from the edge area of ​​the current frame image according to the mask image.

8. A head mounted display device, characterized in that: It includes a processor, a memory, a multi-eye camera and an IMU, wherein the IMU, the multi-eye camera, the memory and the processor are connected via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: Acquire an image sequence collected by a multi-camera, and construct a scale pyramid of a current frame image in the image sequence; the scale pyramid includes images of different scales generated after multiple downsampling of the current frame image; Obtain the 3D coordinates of the first feature point corresponding to the previous frame image, and determine the feature point conversion matrix between the previous frame image and the current frame image in combination with the first IMU pose corresponding to the previous frame image and the second IMU pose corresponding to the current frame image; wherein the first feature point is a feature point extracted from each layer of the image included in the scale pyramid of the previous frame image, and the feature point includes a corner point and an edge point; Determine, according to the feature point conversion matrix and the first feature point corresponding to the previous frame image, a second feature point matched by the first feature point in each layer image included in the scale pyramid of the current frame image; Extracting line features from the current frame image; Obtaining a 3D-2D point matching pair according to the line feature and the second feature point corresponding to the current frame image, and the line feature and the first feature point corresponding to the previous frame image; The initial head pose corresponding to the current frame image is optimized according to the 3D-2D point matching pairs to obtain a target head pose; the initial head pose is determined according to the second IMU pose.

9. The head mounted display device according to claim 8, wherein: The processor extracts line features from the current frame image, and the specific operation is: Acquire a pre-generated mask image having the same pixel size as the current frame image, wherein the mask image includes a non-interested circular region with the center of the image as the origin and a preset number of pixels as the radius, wherein the preset number of pixels is less than the number of pixels of the short side of the current frame image; Line features are extracted from the edge area of ​​the current frame image according to the mask image.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a head-mounted display device to execute the method according to any one of claims 1 to 7.

Images