Near-to-eye display equipment positioning method and device, equipment and storage medium

Abstract

The invention relates to the technical field of positioning, and provides a near-to-eye display equipment positioning method and device, equipment and a storage medium, and the near-to-eye display equipment comprises a reading unit and at least two iteration units. The positioning method comprises the following steps: acquiring a preset positioning matrix of the near-to-eye display equipment, wherein the preset positioning matrix comprises a plurality of columns of preset column elements; controlling a reading unit to read preset column elements of a preset column number from a preset positioning matrix according to the number of the iteration units; the iteration unit and a preset column element have a corresponding relation; according to a preset iteration sequence, an iteration unit is controlled to execute iteration operation on the corresponding preset column elements, and target column elements are obtained; determining a target positioning matrix of the near-to-eye display equipment according to the target column elements; and determining the positioning information of the near-eye display equipment according to the target positioning matrix so as to improve the decomposition efficiency of the target positioning matrix and reduce the access times of the memory space for caching the target positioning matrix, thereby improving the positioning convenience of the near-eye display equipment.

Description

Technical Field

[0001] The present application relates to the field of positioning technology, and in particular to a positioning method, apparatus, device and storage medium for a near-eye display device. Background Art

[0002] At present, Simultaneous Localization and Mapping (SLAM) is widely used in near-eye display devices to locate near-eye display devices. When locating near-eye display devices through SLAM, the nonlinear optimization involved in SLAM usually needs to be iteratively solved through the Gauss-Newton algorithm or the Levenberg-Marquardt (LM) algorithm. These algorithms require the calculation of the Jacobian matrix in each iteration. The Jacobian matrix can be used as a positioning matrix for near-eye display devices to determine the positioning information of near-eye display devices.

[0003] For example, the Jacobian matrix is ​​calculated by the orthogonal triangular decomposition (QR) algorithm. In the related art, the Jacobian matrix can be calculated using the QR decomposition based on the two-dimensional pulse array or the QR decomposition based on the Givens Rotation. However, when the matrix dimension of the Jacobian matrix is ​​large, it is necessary to perform QR decomposition of the Jacobian matrix through more two-dimensional pulse arrays. When the two-dimensional pulse array has high hardware requirements for the near-eye display device, and the hardware of the near-eye display device is limited, it is easy to cause the decomposition efficiency of the Jacobian matrix to be poor. When the matrix dimension of the Jacobian matrix is ​​large, the QR decomposition based on Givens Rotation needs to access the memory space where the Jacobian matrix is ​​cached multiple times, which is easy to cause an increase in the consumption of memory bandwidth. Based on the poor efficiency of Jacobian decomposition and the increase in the consumption of memory bandwidth, it is easy to have an adverse effect on the convenience of positioning of the near-eye display device. Summary of the invention

[0004] The main purpose of the present application is to provide a positioning method, apparatus, device and storage medium for a near-eye display device, aiming to solve the technical problems of poor convenience in positioning the near-eye display device, such as poor efficiency in decomposing a target positioning matrix of the near-eye display device due to the hardware limitations of the near-eye display device, and increased consumption of the memory bandwidth of the near-eye display device due to the need to access the memory space that caches the target positioning matrix multiple times.

[0005] In a first aspect, the present application provides a positioning method for a near-eye display device, the near-eye display device comprising a reading unit and at least two iteration units, the positioning method comprising:

[0006] Acquire a preset positioning matrix of a near-eye display device, wherein the preset positioning matrix includes a plurality of columns of preset column elements;

[0007] According to the number of the iteration units, the reading unit is controlled to read a preset number of column elements from the preset positioning matrix; there is a corresponding relationship between the iteration units and the preset column elements;

[0008] According to the preset iteration order corresponding to the at least two iteration units, control the iteration units to perform iteration operations on the corresponding preset column elements to obtain target column elements;

[0009] Determining a target positioning matrix of the near-eye display device according to the target column elements;

[0010] The positioning information of the near-eye display device is determined according to the target positioning matrix.

[0011] In a second aspect, the present application provides a positioning device for a near-eye display device, the positioning device comprising:

[0012] A preset matrix acquisition module, used to acquire a preset positioning matrix of a near-eye display device, wherein the preset positioning matrix includes a plurality of columns of preset column elements;

[0013] An element reading module, used for controlling a reading unit to read a preset number of column elements from the preset positioning matrix according to the number of iteration units; the iteration units and the preset column elements have a corresponding relationship;

[0014] An iteration module, used for controlling the iteration units to perform iteration operations on the corresponding preset column elements according to the preset iteration orders corresponding to at least two iteration units, so as to obtain target column elements;

[0015] A target matrix determination module, used to determine a target positioning matrix of the near-eye display device according to the target column elements;

[0016] A positioning module is used to determine the positioning information of the near-eye display device according to the target positioning matrix.

[0017] In a third aspect, the present application provides a near-eye display device, the near-eye display device comprising a memory and a processor;

[0018] The memory is used to store computer programs;

[0019] The processor is used to execute the computer program and implement the steps of the positioning method of the near-eye display device as described above when executing the computer program.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the positioning method of the near-eye display device as described above are implemented.

[0021] The present application provides a positioning method, device, equipment and storage medium for a near-eye display device. The near-eye display device includes a reading unit and at least two iteration units. The positioning method includes: obtaining a preset positioning matrix of the near-eye display device, the preset positioning matrix includes multiple columns of preset column elements; according to the number of iteration units, controlling the reading unit to read preset column elements of a preset number of columns from the preset positioning matrix; there is a corresponding relationship between the iteration unit and the preset column element; according to the preset iteration order corresponding to at least two iteration units, controlling the iteration unit to perform an iteration operation on the corresponding preset column element to obtain a target column element; according to the target column element, determining the target positioning matrix of the near-eye display device; according to the target positioning matrix, determining the positioning information of the near-eye display device.

[0022] When the preset positioning matrix of the near-eye display device is obtained, the preset positioning matrix may include a Jacobian matrix, and it is necessary to perform QR decomposition on the preset positioning matrix to determine the target positioning matrix. In the case where the near-eye display device includes at least two iteration units, the near-eye display device can control the at least two iteration units to perform iterative operations on the corresponding preset column elements in the preset positioning matrix in parallel according to the preset iteration order corresponding to the at least two iteration units, so as to determine the target positioning matrix corresponding to the preset positioning matrix. Since the iteration unit has low hardware requirements for the near-eye display device, it is beneficial to improve the decomposition efficiency of the preset positioning matrix by performing iterative operations on the preset positioning matrix in parallel through at least two iteration units. Accordingly, the reading unit can read the preset column elements of the preset number of columns required by the at least two iteration units from the preset positioning matrix by a single access to the memory space where the preset positioning matrix is ​​located, which is beneficial to reduce the consumption of the memory bandwidth of the near-eye display device. In the case of determining the target positioning matrix, the target positioning matrix can be used to determine the positioning information of the near-eye display device, which is beneficial to improve the convenience of positioning the near-eye display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 It is a flowchart of a positioning method for a near-eye display device provided in an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of a preset positioning matrix involved in an embodiment of the present application;

[0026] Figure 3 It is a schematic diagram of an iterative operation of an iterative unit on preset column elements according to an embodiment of the present application;

[0027] Figure 4 It is a flowchart of iterative operation of an iterative unit on preset column elements involved in an embodiment of the present application;

[0028] Figure 5 is a schematic block diagram of a positioning device for a near-eye display device provided in an embodiment of the present application;

[0029] Figure 6 It is a schematic block diagram of a near-eye display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0032] The embodiments of the present application provide a positioning method, device, equipment and storage medium for a near-eye display device. The positioning method for a near-eye display device can be applied to a near-eye display device. A near-eye display device may include augmented reality (AR) glasses, mixed reality (MR) glasses, AR helmets, MR helmets, etc., without limitation.

[0033] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0034] See also Figure 1 , Figure 1 It is a flow chart of a method for positioning a near-eye display device provided in an embodiment of the present application. It should be noted that the method for positioning a near-eye display device provided in an embodiment of the present application can be used in a near-eye display device, and is not limited here.

[0035] like Figure 1 As shown, the positioning method of the near-eye display device includes steps S101 to S105.

[0036] Exemplarily, the near-eye display device includes a reading unit and at least two iteration units.

[0037] For example, the reading unit may be used to read a preset column element included in a preset positioning matrix of the near-eye display device. The iteration unit may be used to perform an iterative operation on the preset column element read by the reading unit. The near-eye display device may be provided with at least two iteration units based on the original QR decomposition based on Givens Rotation to perform QR decomposition on the preset positioning matrix of the near-eye display device to determine the positioning information of the near-eye display device.

[0038] S101. Obtain a preset positioning matrix of a near-eye display device, where the preset positioning matrix includes a plurality of columns of preset column elements.

[0039] For example, the near-eye display device may collect sensor data required for positioning the near-eye display device. When the sensor data is available for determining the positioning information of the near-eye display device, the preset positioning matrix of the near-eye display device may be determined based on the sensor data. Accordingly, based on the matrix properties of the preset positioning matrix, the preset positioning matrix may include multiple columns of preset column elements.

[0040] In some embodiments, a preset positioning matrix is ​​determined based on image sensing data and device posture sensing data collected by the near-eye display device.

[0041] For example, an image sensor and an inertial measurement unit (IMU) are provided on the near-eye display device. The near-eye display device can collect image sensing data of the environment in which the near-eye display device is located through the image sensor. The near-eye display device can collect device posture sensing data of the near-eye display device through the IMU. The device posture sensing data may include the position and direction of the near-eye display device. Since both the image sensing data and the device posture sensing data can be used to determine the positioning information of the near-eye display device, the near-eye display device can combine the image sensing data and the device posture sensing data to determine a preset positioning matrix. The preset positioning matrix can be used to determine the target positioning matrix of the near-eye display device. The target positioning matrix can be used to determine the positioning information of the near-eye display device.

[0042] When the preset positioning matrix is ​​determined, the positioning information of the near-eye display device can be subsequently determined by means of matrix operations, which is helpful to improve the convenience of positioning the near-eye display device.

[0043] In some embodiments, feature extraction and feature matching are performed on image sensor data to obtain a first feature; integration processing is performed on device posture sensor data to obtain a second feature; and a preset positioning matrix is ​​determined based on the first feature and the second feature.

[0044] For example, a near-eye display device can use a visual-inertial odometry (VIO) or a visual-inertial system (VINS) to fuse the image sensor data collected by the near-eye display device and the device posture data to achieve SLAM.

[0045] like Figure 2 As shown, the near-eye display device can perform feature extraction and feature matching processing on the image sensing data to obtain a first feature. The near-eye display device can perform integration processing on the device posture sensing data to obtain a second feature. The near-eye display device can perform fusion processing on the first feature and the second feature to obtain a preset positioning matrix. For example, the preset positioning matrix may include an H matrix of a state equation. The H matrix can be expressed as:

[0046] H=[0 2M×9 H p [H ω1:-1 0 2M×9 H ωN 0 2M×9 ]]

[0047] Among them, M is used to indicate the dimension of the first feature, 0 2M×9 , H p , H ω1:-1 and H ωN Used to determine the row and column dimensions in the H matrix, 0 2M×9 It is used to indicate the first state matrix corresponding to the H matrix, H p It is used to indicate the second state matrix corresponding to the first feature and the second feature, H ω1:-1 It is used to indicate the third state matrix corresponding to the ω1th row to the N-1th row of the H matrix, H ωN It is used to indicate the fourth state matrix corresponding to the ωNth row of the H matrix. The row dimension is the dimension of the observation residuals corresponding to all first features, and the column dimension is the dimension of the state quantity of all frames in the window corresponding to the first feature.

[0048] The typical value of M is 100×11, that is, the window corresponding to the first feature is 11, and 100 first features can be observed in each window, 0 2M×9 , H p , H ω1:-1 and H ωN The corresponding row dimensions are 2200, 02M×9 The corresponding column dimension is 9, H p The corresponding column dimension is 7, H ω1:-1 The column dimension of the first 10 rows in the corresponding window is 6, H ωN Taking the last row of the corresponding window as 15 as an example, the dimension of the H matrix can be determined as: 2200×(9+7+10×6+9+15+9)=2200×109. Among them, 2200 is used to indicate the row dimension of the H matrix, and 109 is used to indicate the column dimension of the H matrix. Of course, the preset positioning matrix is ​​not limited to the H matrix, and the H matrix is ​​not limited to this, and no limitation is made here.

[0049] Accordingly, since the dimension of the H matrix is ​​large, the H matrix can be QR-decomposed to solve the nonlinear optimization problem and obtain the target positioning matrix. According to the target positioning matrix, the optimal estimate of the state can be determined. The optimal estimate of the state can be used to indicate the optimal estimate of the position, direction and speed of the near-eye display device, which can be used to determine the positioning information of the near-eye display device.

[0050] In this way, by obtaining the preset positioning matrix of the near-eye display device, the near-eye display device can subsequently perform QR decomposition on the preset positioning matrix to determine the corresponding target positioning matrix, thereby determining the positioning information of the near-eye display device.

[0051] S102. According to the number of iteration units, control the reading unit to read a preset number of preset column elements from a preset positioning matrix; there is a corresponding relationship between the iteration unit and the preset column elements.

[0052] For example, when the near-eye display device includes at least two iteration units, the near-eye display device can perform QR decomposition on the preset positioning matrix in parallel through the at least two iteration units to improve the decomposition efficiency of the preset positioning matrix. Based on this, the near-eye display device can control the reading unit to read the preset column elements of the preset number of columns from the preset positioning matrix according to the number of iteration units.

[0053] For example, the iteration units include PE0, PE1, and PE2, and the number of iteration units is 3. The near-eye display device performs QR decomposition on the preset positioning matrix in parallel through PE0 to PE2.

[0054] Taking the preset positioning matrix as a matrix A with a dimension of 9×9 as an example, the matrix A can be expressed as:

[0055]

[0056] The digital identifiers of the elements included in each preset column element are used to indicate the row and column numbers of the elements in matrix A. For example, element x11 is located in the 1st row and 1st column of matrix A, element x21 is located in the 2nd row and 1st column of matrix A, and so on.

[0057] When the number of iteration units is 3, the near-eye display device can control the reading unit to read the preset column elements of the 1st to 3rd columns of the matrix A. Correspondingly, the near-eye display device can set PE0 to correspond to the preset column elements of the 1st column, PE1 to correspond to the preset column elements of the 2nd column, and PE2 to correspond to the preset column elements of the 3rd column. Of course, it is not limited to this. The subsequent near-eye display device can also control the reading unit to read the preset column elements of the 4th to 6th columns of the matrix A, and then control the reading unit to read the preset column elements of the 7th to 9th columns of the matrix A. Based on this, when the number of iteration units includes 3, the reading unit, for example, needs to access the memory space cached with matrix A 3 times to complete the reading operation of all preset column elements included in matrix A. However, based on the original QR decomposition based on Givens Rotation, its reading unit needs to read the preset column elements included in a column of matrix A each time, so its reading unit, for example, needs to access the memory space cached with matrix A 9 times to complete the reading operation of all preset column elements included in matrix A.

[0058] In this way, when the reading unit reads the preset column elements of the preset number of columns from the preset positioning matrix, the reading unit can read the preset column elements of the preset number of columns by accessing the memory space where the preset positioning matrix is ​​located once, which is conducive to reducing the number of accesses to the memory space, thereby helping to reduce the consumption of the memory bandwidth of the near-eye display device. Accordingly, the reading unit can subsequently provide the preset column elements to the corresponding iteration units according to the corresponding relationship between the iteration units and the preset column elements, so that at least two iteration units perform corresponding iteration operations on the preset column elements corresponding to each other in parallel, which is conducive to subsequently improving the decomposition efficiency of the preset positioning matrix.

[0059] In some implementations, according to the number of iteration units, the reading unit is controlled to read a preset number of column elements from the preset positioning matrix from left to right.

[0060] Take the preset positioning matrix as matrix A, and the iteration units include PE0 to PE2 as an example. The near-eye display device can determine the number of columns corresponding to each preset column element in matrix A from left to right. For example, the near-eye display device can determine the leftmost preset column element in matrix A as the first column preset column element, the rightmost preset column element in matrix A as the ninth column preset column element, and so on, to determine the order of the preset column elements between the first column and the ninth column preset column elements in matrix A.

[0061] When the number of iteration units is 3, such as PE0 to PE2, the near-eye display device can control the reading unit to read 3 columns of preset column elements from the matrix A from left to right, such as the 1st to 3rd columns of preset column elements. The 1st to 3rd columns of preset column elements read by the reading unit can be used for subsequent PE0 to PE2 to perform iteration operations on the corresponding 1st to 3rd columns of preset column elements.

[0062] Of course, it is not limited to this. When PE0 to PE2 complete the iteration operation on the corresponding preset column elements of the 1st to 3rd columns respectively, the reading unit can continue to read the preset column elements of the 4th to 6th columns from the matrix A from left to right according to the number of iteration units, such as the number of iteration units corresponding to PE0 to PE2 is 3. Similarly, when PE0 to PE2 complete the iteration operation on the corresponding preset column elements of the 4th to 6th columns respectively, the reading unit can continue to read the preset column elements of the 7th to 9th columns from the matrix A from left to right.

[0063] In this way, the near-eye display device can ensure that the reading unit regularly reads the preset column elements in the preset positioning matrix by controlling the reading unit to read the preset column elements of the preset number of columns from the preset positioning matrix in the direction from left to right according to the number of iteration units, and then the subsequent reading unit can provide the corresponding preset column elements to the corresponding iteration unit for iteration operation to obtain the target column elements. Accordingly, when the target positioning matrix is ​​determined according to the target column elements, it is helpful to improve the convenience of determining the position of the target column elements in the target positioning matrix.

[0064] S103. According to the preset iteration order corresponding to at least two iteration units, control the iteration units to perform iteration operations on the corresponding preset column elements to obtain target column elements.

[0065] In some embodiments, when controlling the iteration unit to perform an iteration operation on the corresponding preset column element, the near-eye display device may control the iteration unit to perform two iteration operations on the preset column element to obtain the target column element.

[0066] For example, in the case where the near-eye display device includes at least two iteration units, there is a corresponding preset iteration order for the at least two iteration units. Take the iteration units including PE0 to PE2 as an example. The preset iteration order indicates that the iteration order of PE0 is before the iteration order of PE1, and the iteration order of PE1 is before the iteration order of PE2. Based on the indication of the preset iteration order, the near-eye display device can preferentially control PE0 to perform an iteration operation on the preset column element corresponding to PE0, and obtain the target column element corresponding to PE0. After PE0 completes the iteration operation on the preset column element corresponding to PE0, the near-eye display device can control PE1 to perform an iteration operation on the preset column element corresponding to PE1, and obtain the target column element corresponding to PE1. Similarly, after PE1 completes the iteration operation on the preset column element corresponding to PE1, the near-eye display device can control PE2 to perform an iteration operation on the preset column element corresponding to PE2, and obtain the target column element corresponding to PE2.

[0067] In some embodiments, the iteration unit includes n iteration units, and a preset iteration order indicates that the iteration order of the i-th iteration unit among the n iteration units is before the iteration order of the j-th iteration unit, and j is greater than i; the preset column elements of the preset column number include n-column preset column elements, and the i-th column preset column element among the n-column preset column elements is located to the left of the j-th column preset column element among the n-column preset column elements; the i-th column preset column element corresponds to the i-th iteration unit, and the j-th column preset column element corresponds to the j-th iteration unit.

[0068] For example, when the number of iteration units is n, the near-eye display device can control the reading unit to read n columns of preset column elements from the preset positioning matrix in a left-to-right direction. Accordingly, the near-eye display device can determine that the i-th iteration unit corresponds to the i-th column element, and the j-th iteration unit corresponds to the j-th column element. Since the preset iteration order indicates that the iteration order of the i-th iteration unit is before the iteration order of the j-th iteration unit, the near-eye display device can first control the i-th iteration unit to perform an iteration operation on the i-th column element. After the i-th iteration unit completes the iteration operation on the i-th column element, the near-eye display device controls the j-th iteration unit to perform an iteration operation on the j-th column element. And so on, until the iteration operation on all preset column elements in the preset positioning matrix is ​​completed, and the target column elements corresponding to each preset column element are obtained.

[0069] For example, the preset positioning matrix includes matrix A, and the iteration units include PE0 to PE2. Figure 3As shown, the near-eye display device can divide the preset column elements included in the matrix A into the 1st to 9th columns of preset column elements from left to right. In the first round of iteration, the reading unit can provide the preset column elements from the 1st to 3rd columns read from the matrix A to PE0 to PE2 respectively. The near-eye display device can first control PE0 to complete the iterative operation on the preset column elements in the 1st column, then control PE1 to complete the iterative operation on the preset column elements in the 2nd column, and finally control PE2 to complete the iterative operation on the preset column elements in the 3rd column. In the second round of iteration, the reading unit can provide the preset column elements from the 4th to 6th columns read from the matrix A to PE0 to PE2 respectively. The near-eye display device can first control PE0 to complete the iterative operation on the preset column elements in the 4th column, then control PE1 to complete the iterative operation on the preset column elements in the 5th column, and finally control PE2 to complete the iterative operation on the preset column elements in the 6th column. In the third round of iteration, the reading unit can provide the preset column elements from the 7th to 9th columns read from the matrix A to PE0 to PE2 respectively. The near-eye display device may first control PE0 to complete the iteration operation on the preset column elements of the 7th column, then control PE1 to complete the iteration operation on the preset column elements of the 8th column, and finally control PE2 to complete the iteration operation on the preset column elements of the 9th column. When PE2 completes the iteration operation on the preset column elements of the 9th column, each column of preset column elements included in the matrix A completes the iteration, and the near-eye display device may determine the target column elements corresponding to each of the preset column elements in the matrix A. The target column elements may be used to determine the target positioning matrix of the near-eye display device to determine the positioning information of the near-eye display device.

[0070] In this way, when the near-eye display device controls the iteration unit to perform an iterative operation on the corresponding preset column elements according to the preset iteration order corresponding to at least two iteration units to obtain the target column elements, since the near-eye display device can control at least two iteration units to perform iterative operations on the preset column elements corresponding to each iteration unit in parallel according to the preset iteration order, it is beneficial to improve the decomposition efficiency of the preset positioning matrix. Accordingly, since the target column elements can be used to determine the target positioning matrix of the near-eye display device to determine the positioning information of the near-eye display device, the determination of the target column elements is beneficial to improve the convenience of positioning the near-eye display device.

[0071] Exemplarily, the preset column element includes multiple rows of preset row elements. For example, based on the matrix property of the preset positioning matrix, for the multiple columns of preset column elements included in the preset positioning matrix, the total number of rows corresponding to the preset row elements included in different columns of preset column elements is the same.

[0072] In some embodiments, when the control reading unit reads the preset column elements, if the total number of rows corresponding to multiple rows of preset row elements is greater than a preset row number threshold, the control reading unit reads the preset row elements of the preset number of rows included in the preset column elements from the preset positioning matrix in a bottom-to-top direction; the preset number of rows is less than or equal to the preset row number threshold.

[0073] For example, the preset row number threshold may be determined based on the spatial size of the cache space of the reading unit. In the case where the spatial size of the cache space of the reading unit is limited, it is easy for the reading unit to be unable to directly read all the preset row elements included in the preset column elements of the preset number of columns when reading the preset column elements of the preset number of columns. Based on this, the near-eye display device may control the reading unit to read all the preset row elements included in the preset column elements of the preset number of columns in multiple times. In some embodiments, the near-eye display device may control the reading unit to read the preset row elements of the preset number of rows included in the preset column elements from the preset positioning matrix in a bottom-to-top direction. For example, the near-eye display device may determine the preset row element at the bottom of the preset column element as the first row of preset row elements, and determine the preset row element at the top of the preset column element as the last row of preset row elements, and so on, to determine the order of the preset row elements between the first row and the last row of preset row elements.

[0074] like Figure 4 As shown, the near-eye display device controls the reading unit to obtain the preset row elements of the preset number of rows included in the preset column elements. Take the example that the iteration unit includes PE0 to PE2, the preset column elements of the preset number of columns include the preset column elements of the 1st to 3rd columns, the total number of rows corresponding to the preset row elements is 9, and the preset row number threshold is 5. The preset number of rows may include any one of 1 to 5. The preset number of rows of the preset row elements read by the reading unit each time may be the same or different. For example, the reading unit may read the 3 rows of preset row elements respectively included in the preset column elements of the 1st to 3rd columns in sequence until all the preset row elements respectively included in the preset column elements of the 1st to 3rd columns are read. For another example, the reading unit may first read the 5 rows of preset row elements respectively included in the preset column elements of the 1st to 3rd columns, and then read the 4 rows of preset row elements respectively included in the preset column elements of the 1st to 3rd columns. No limitation is made here.

[0075] Correspondingly, the near-eye display device may subsequently provide the preset number of row elements included in the preset column elements read by the reading unit to the corresponding iteration unit to determine the target column element corresponding to the preset column element.

[0076] In this way, when the size of the cache space of the reading unit is limited, the near-eye display device can flexibly control the reading unit to read the preset row elements of the preset number of rows included in the preset column elements based on the size of the cache space of the reading unit, which is conducive to improving the reading flexibility of the reading unit for the preset column elements.

[0077] In some implementations, the control iteration unit performs an iteration operation on a preset number of row elements until the iteration operation on all preset row elements is completed to obtain a target column element.

[0078] like Figure 4 As shown, the near-eye display device controls the iteration unit to perform an iteration operation on the preset row elements of the preset number of rows. Take the example that the iteration unit includes PE0 to PE2, the preset positioning matrix includes the preset column elements of the 1st to 3rd columns, and the preset number of rows is 3. The near-eye display device can provide the preset row elements of the 1st to 3rd rows respectively included in the preset column elements of the 1st to 3rd columns read by the reading unit to PE0 to PE2. The near-eye display device can first control PE0 to complete the iteration operation on the preset row elements of the 1st to 3rd rows included in the preset column elements of the 1st column, and then control PE1 to complete the iteration operation on the preset row elements of the 1st to 3rd rows included in the preset column elements of the 2nd column, and finally control PE2 to complete the iteration operation on the preset row elements of the 1st to 3rd rows included in the preset column elements of the 3rd column.

[0079] like Figure 4 As shown, the near-eye display device writes the preset row elements of the preset number of rows after the iterative processing is completed back to the reading unit. For example, the near-eye display device may write the preset row elements of the first row to the third row included in the preset column elements of the first column after the iterative processing is completed back to the reading unit. The near-eye display device may also write the preset row elements of the first row to the third row included in the preset column elements of the second column after the iterative processing is completed back to the reading unit, and so on.

[0080] like Figure 4As shown, in the case where the iteration unit has not completed the iteration operation on all preset row elements included in the corresponding preset column element, the near-eye display device can cyclically control the reading unit to read the preset row elements of the preset number of rows included in the preset column element, control the iteration unit to perform the iteration operation on the preset row elements read by the reading unit, and write the preset row elements of the preset number of rows after the iteration processing back to the reading unit, until the iteration unit completes the iteration operation on all preset row elements included in the preset column element and obtains the target column element. For example, in the case where PE0 to PE2 complete the iteration operation on the preset row elements of the preset number of rows included in the preset column elements corresponding to each of them, the near-eye display device can provide the preset row elements of the 4th row to the preset row elements of the 6th row included in the preset column elements of the 1st column to the preset column elements of the 3rd column read by the reading unit to PE0 to PE2 respectively, and perform the iteration operation on the preset row elements of the 4th row to the preset row elements of the 6th row included in the corresponding preset column elements according to the preset iteration order corresponding to PE0 to PE2. And so on, until PE0 completes the iterative operation on all preset row elements included in the preset column elements in the first column, and obtains the target column elements in the first column corresponding to the preset column elements in the first column, PE1 completes the iterative operation on all preset row elements included in the preset column elements in the second column, and obtains the target column elements in the second column corresponding to the preset column elements in the second column, and PE2 completes the iterative operation on all preset row elements included in the preset column elements in the third column, and obtains the target column elements in the third column corresponding to the preset column elements in the third column.

[0081] In this way, when the total number of rows corresponding to the preset row elements is greater than the preset row number threshold, the near-eye display device controls the iteration unit to perform an iteration operation on the preset row elements of the preset number of rows until the iteration operation on all preset row elements is completed to obtain the target column elements. This is beneficial for improving the iteration flexibility of the iteration unit on the preset column elements when the cache space of the reading unit is limited.

[0082] Based on the improvement of the reading flexibility of the reading unit for the preset column elements and the improvement of the iteration flexibility of the iteration unit for the preset column elements, it is beneficial to improve the flexibility of the near-eye display device in decomposing the preset positioning matrix, thereby helping to improve the decomposition efficiency of the preset positioning matrix.

[0083] S104: Determine a target positioning matrix of the near-eye display device according to the target column elements.

[0084] For example, when determining the target column element corresponding to the preset column element, the near-eye display device may determine the position of the target column element in the target positioning matrix.

[0085] In some embodiments, the near-eye display device may determine the position of the target column element corresponding to the preset column element in the target positioning matrix according to the position of the preset column element in the preset positioning matrix. Take the preset positioning matrix including matrix A as an example. When the near-eye display device determines the target column elements of the first to ninth columns corresponding to the preset column elements of the first to ninth columns in matrix A, the position of each of the target column elements of the first to ninth columns in the target positioning matrix may be determined from left to right, thereby determining the target positioning matrix corresponding to matrix A.

[0086] In some embodiments, since the target column elements are obtained by the near-eye display device according to the preset iteration order corresponding to at least two iteration units, and the control iteration unit performs an iteration operation on the corresponding preset column elements, the target column elements of multiple columns may have a corresponding determination order. Take the preset positioning matrix including matrix A as an example. The near-eye display device determines the order of the first column target column elements corresponding to the first column preset column elements earlier than the second column target column elements corresponding to the second column preset column elements. Correspondingly, the near-eye display device determines the order of the second column target column elements corresponding to the second column preset column elements earlier than the third column target column elements corresponding to the third column preset column elements. By analogy, the determination order of the target column elements corresponding to the first to ninth columns of the preset column elements included in matrix A can be determined. Accordingly, the near-eye display device can determine the positions of the first to ninth columns of the target column elements in the target positioning matrix according to the determination order of the target column elements corresponding to the first to ninth columns of the preset column elements, from left to right, thereby determining the target positioning matrix corresponding to matrix A.

[0087] In this way, the near-eye display device determines the target positioning matrix of the near-eye display device according to the target column elements. The target positioning matrix can be used to determine the positioning information of the near-eye display device, which is conducive to improving the convenience of positioning the near-eye display device.

[0088] S105. Determine positioning information of the near-eye display device according to the target positioning matrix.

[0089] For example, when the target positioning matrix is ​​determined, the near-eye display device can simplify the computational complexity of determining the positioning information of the near-eye display device through the target positioning matrix. In addition, the target positioning matrix can be used to determine the positioning information of the near-eye display device, which is conducive to improving the convenience of positioning the near-eye display device.

[0090] The positioning method of the near-eye display device provided in the above embodiment includes: obtaining a preset positioning matrix of the near-eye display device, the preset positioning matrix including multiple columns of preset column elements; according to the number of iteration units, controlling the reading unit to read a preset number of preset column elements from the preset positioning matrix; there is a corresponding relationship between the iteration unit and the preset column elements; according to a preset iteration order corresponding to at least two iteration units, controlling the iteration unit to perform an iteration operation on the corresponding preset column elements to obtain a target column element; according to the target column elements, determining a target positioning matrix of the near-eye display device; according to the target positioning matrix, determining the positioning information of the near-eye display device.

[0091] When the preset positioning matrix of the near-eye display device is obtained, the preset positioning matrix may include a Jacobian matrix, and it is necessary to perform QR decomposition on the preset positioning matrix to determine the target positioning matrix. In the case where the near-eye display device includes at least two iteration units, the near-eye display device can control the at least two iteration units to perform iterative operations on the corresponding preset column elements in the preset positioning matrix in parallel according to the preset iteration order corresponding to the at least two iteration units, so as to determine the target positioning matrix corresponding to the preset positioning matrix. Since the iteration unit has low hardware requirements for the near-eye display device, it is beneficial to improve the decomposition efficiency of the preset positioning matrix by performing iterative operations on the preset positioning matrix in parallel through at least two iteration units. Accordingly, the reading unit can read the preset column elements of the preset number of columns required by the at least two iteration units from the preset positioning matrix by a single access to the memory space where the preset positioning matrix is ​​located, which is beneficial to reduce the consumption of the memory bandwidth of the near-eye display device. In the case of determining the target positioning matrix, the target positioning matrix can be used to determine the positioning information of the near-eye display device, which is beneficial to improve the convenience of positioning the near-eye display device.

[0092] See also Figure 5 , Figure 5 1 is a schematic block diagram of a positioning device for a near-eye display device provided in an embodiment of the present application. The positioning device can be configured in a near-eye display device to execute the positioning method for the near-eye display device described above. The near-eye display device can include AR glasses, VR glasses, AR helmets, VR helmets, etc., which are not limited here.

[0093] like Figure 5 As shown, the positioning device includes a preset matrix acquisition module 110 , an element reading module 120 , an iteration module 130 , a target matrix determination module 140 and a positioning module 150 .

[0094] The preset matrix acquisition module 110 is used to acquire a preset positioning matrix of a near-eye display device, where the preset positioning matrix includes a plurality of columns of preset column elements.

[0095] The element reading module 120 is used to control the reading unit to read a preset number of column elements from the preset positioning matrix according to the number of iteration units; there is a corresponding relationship between the iteration unit and the preset column elements.

[0096] The iteration module 130 is used to control the iteration units to perform iteration operations on the corresponding preset column elements according to the preset iteration orders corresponding to at least two iteration units to obtain target column elements.

[0097] The target matrix determination module 140 is used to determine the target positioning matrix of the near-eye display device according to the target column elements.

[0098] The positioning module 150 is used to determine the positioning information of the near-eye display device according to the target positioning matrix.

[0099] Exemplarily, the element reading module 120 includes a first element reading submodule.

[0100] The first element reading submodule is used to control the reading unit to read a preset number of column elements from the preset positioning matrix from left to right according to the number of the iteration units.

[0101] Exemplarily, the iteration module 130 includes a first iteration submodule.

[0102] A first iteration submodule, for the iteration units including n iteration units, the preset iteration order indicates that the iteration order of the i-th iteration unit among the n iteration units is before the iteration order of the j-th iteration unit, and j is greater than i; the preset column elements of the preset column number include n-column preset column elements, the i-column preset column element among the n-column preset column elements is located to the left of the j-column preset column element among the n-column preset column elements; the i-column preset column element corresponds to the i-th iteration unit, and the j-column preset column element corresponds to the j-th iteration unit.

[0103] Exemplarily, the preset column elements include a plurality of rows of preset row elements.

[0104] The first element reading submodule includes a second element reading submodule.

[0105] The second element reading submodule is used to control the reading unit to read the preset column element, if the total number of rows corresponding to the multiple rows of preset row elements is greater than a preset row number threshold, to control the reading unit to read the preset row elements of the preset number of rows included in the preset column element from the preset positioning matrix in a bottom-to-top direction; the preset number of rows is less than or equal to the preset row number threshold.

[0106] Exemplarily, the first iterative submodule includes the second iterative submodule.

[0107] The second iteration submodule is used to control the iteration unit to perform iteration operations on the preset row elements of a preset number of rows until the iteration operations on all the preset row elements are completed to obtain the target column elements.

[0108] Exemplarily, the preset matrix acquisition module 110 includes a preset matrix determination submodule.

[0109] The preset matrix determination submodule is used to determine the preset positioning matrix according to the image sensing data and the device posture sensing data collected by the near-eye display device.

[0110] Exemplarily, the preset matrix determination submodule includes a first feature determination submodule, a second feature determination submodule and a feature fusion submodule.

[0111] The first feature determination submodule is used to perform feature extraction and feature matching processing on the image sensing data to obtain a first feature.

[0112] The second feature determination submodule is used to perform integration processing on the device posture sensor data to obtain the second feature.

[0113] A feature fusion submodule is used to determine the preset positioning matrix according to the first feature and the second feature.

[0114] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0115] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0116] Exemplarily, the above method and apparatus may be implemented in the form of a computer program, which may be run on a near-eye display device to locate the near-eye display device. Exemplarily, the near-eye display device may include VR glasses, AR glasses, VR helmets, AR helmets, etc., which are not limited here.

[0117] See also Figure 6 , Figure 6 It is a schematic block diagram of the structure of a near-eye display device provided in an embodiment of the present application.

[0118] like Figure 6 As shown, the near-eye display device includes a memory and a processor. The memory and the processor may be connected via a system bus, and the memory may include a storage medium and an internal memory.

[0119] The storage medium may store an operating system and a computer program. When the computer program is executed, the processor may execute any positioning method of a near-eye display device.

[0120] The processor is used to provide computing and control capabilities to support the operation of the entire near-eye display device.

[0121] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any positioning method of the near-eye display device.

[0122] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a portion of the structure related to the scheme of the present application, and does not constitute a limitation on the near-eye display device to which the scheme of the present application is applied. The specific near-eye display device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0124] In one embodiment, the processor is used to execute a computer program and implement the following steps when executing the computer program:

[0125] Acquire a preset positioning matrix of a near-eye display device, wherein the preset positioning matrix includes a plurality of columns of preset column elements;

[0126] According to the number of the iteration units, the reading unit is controlled to read a preset number of column elements from the preset positioning matrix; there is a corresponding relationship between the iteration units and the preset column elements;

[0127] According to the preset iteration order corresponding to the at least two iteration units, control the iteration units to perform iteration operations on the corresponding preset column elements to obtain target column elements;

[0128] Determining a target positioning matrix of the near-eye display device according to the target column elements;

[0129] The positioning information of the near-eye display device is determined according to the target positioning matrix.

[0130] It should be noted that, those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the positioning of the near-eye display device described above can refer to the corresponding process in the aforementioned positioning method embodiment of the near-eye display device, and will not be repeated here.

[0131] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The method implemented when the computer program is executed by a processor can refer to the various embodiments of the positioning method of the near-eye display device of the present application.

[0132] The computer-readable storage medium may be an internal storage unit of the near-eye display device described in the aforementioned embodiment, such as a hard disk or memory of the near-eye display device. The computer-readable storage medium may also be an external storage device of the near-eye display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the near-eye display device.

[0133] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0134] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0135] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A positioning method for a near-eye display device, characterized in that: The near-eye display device comprises a reading unit and at least two iteration units; The positioning method comprises: Acquire a preset positioning matrix of a near-eye display device, wherein the preset positioning matrix includes a plurality of columns of preset column elements; According to the number of the iteration units, the reading unit is controlled to read a preset number of column elements from the preset positioning matrix; there is a corresponding relationship between the iteration units and the preset column elements; According to the preset iteration order corresponding to the at least two iteration units, control the iteration units to perform iteration operations on the corresponding preset column elements to obtain target column elements; Determining a target positioning matrix of the near-eye display device according to the target column elements; The positioning information of the near-eye display device is determined according to the target positioning matrix.

2. The positioning method according to claim 1, characterized in that: The step of controlling the reading unit to read a preset number of column elements from the preset positioning matrix according to the number of the iteration units includes: According to the number of the iteration units, the reading unit is controlled to read a preset number of column elements from the preset positioning matrix from left to right.

3. The positioning method according to claim 2, characterized in that: The iteration unit includes n iteration units, and the preset iteration order indicates that the iteration order of the i-th iteration unit among the n iteration units is before the iteration order of the j-th iteration unit, and j is greater than i; the preset column elements of the preset column number include n-column preset column elements, and the i-column preset column element among the n-column preset column elements is located to the left of the j-column preset column element among the n-column preset column elements; the i-column preset column element corresponds to the i-th iteration unit, and the j-column preset column element corresponds to the j-th iteration unit.

4. The positioning method according to claim 2, characterized in that: The preset column elements include a plurality of rows of preset row elements; The controlling the reading unit to read a preset number of preset column elements from the preset positioning matrix from left to right includes: When controlling the reading unit to read the preset column element, if the total number of rows corresponding to the multiple rows of preset row elements is greater than a preset row number threshold, controlling the reading unit to read the preset number of rows of preset row elements included in the preset column element from the preset positioning matrix in a bottom-to-top direction; The preset number of rows is less than or equal to the preset number of rows threshold.

5. The positioning method according to claim 4, characterized in that: The controlling the iteration unit to perform an iteration operation on the corresponding preset column element to obtain a target column element includes: The iteration unit is controlled to perform an iteration operation on the preset row elements of a preset number of rows until the iteration operation on all the preset row elements is completed to obtain a target column element.

6. The positioning method according to any one of claims 1 to 5, characterized in that: The obtaining of a preset positioning matrix of a near-eye display device includes: The preset positioning matrix is ​​determined according to the image sensing data and the device posture sensing data collected by the near-eye display device.

7. The positioning method according to claim 6, characterized in that: The determining the preset positioning matrix according to the image sensor data and the device posture sensor data collected by the near-eye display device includes: Performing feature extraction and feature matching processing on the image sensing data to obtain a first feature; Performing integration processing on the device posture sensor data to obtain a second feature; The preset positioning matrix is ​​determined according to the first feature and the second feature.

8. A positioning device for a near-eye display device, characterized in that: The positioning device comprises: A preset matrix acquisition module, used to acquire a preset positioning matrix of a near-eye display device, wherein the preset positioning matrix includes a plurality of columns of preset column elements; An element reading module, used for controlling a reading unit to read a preset number of column elements from the preset positioning matrix according to the number of iteration units; the iteration units and the preset column elements have a corresponding relationship; An iteration module, used for controlling the iteration units to perform iteration operations on the corresponding preset column elements according to the preset iteration orders corresponding to at least two iteration units, so as to obtain target column elements; A target matrix determination module, used to determine a target positioning matrix of the near-eye display device according to the target column elements; A positioning module is used to determine the positioning information of the near-eye display device according to the target positioning matrix.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the steps of the method for positioning a near-eye display device according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for positioning a near-eye display device according to any one of claims 1 to 7 are implemented.

Images