Positioning methods, devices, equipment, and storage media for near-eye display devices

Abstract

This application relates to the field of positioning technology, and provides a positioning method, apparatus, device, and storage medium for a near-eye display device. The near-eye display device includes a reading unit and at least two iteration units. The positioning method includes: acquiring a preset positioning matrix of the near-eye display device, the preset positioning matrix including multiple preset column elements; controlling the reading unit to read a preset number of preset column elements from the preset positioning matrix according to the number of iteration units; there is a correspondence between the iteration units and the preset column elements; controlling the iteration units to perform iteration operations on the corresponding preset column elements according to a preset iteration order to obtain target column elements; determining the target positioning matrix of the near-eye display device based on the target column elements; and determining the positioning information of the near-eye display device based on the target positioning matrix, so as to improve the decomposition efficiency of the target positioning matrix, reduce the number of accesses to the memory space of the cached target positioning matrix, and thus improve the convenience of positioning the near-eye display device.

Description

Technical Field

[0001] This application relates to the field of positioning technology, and in particular to a positioning method, apparatus, device and storage medium for a near-eye display device. Background Technology

[0002] Currently, Simultaneous Localization and Mapping (SLAM) is widely used in near-eye display devices for localization. When using SLAM to localize near-eye display devices, the nonlinear optimization involved typically requires iterative solutions using Gauss-Newton or Levenberg-Marquardt (LM) algorithms. These algorithms require calculating the Jacobian matrix in each iteration. The Jacobian matrix can serve as the localization matrix for the near-eye display device, determining its location information.

[0003] For example, the Jacobian matrix is ​​calculated using the orthogonal triangular decomposition (QR) algorithm. Related technologies may employ QR decomposition based on two-dimensional pulse arrays or Givens Rotation. However, when the Jacobian matrix has a large dimension, multiple two-dimensional pulse arrays are required for QR decomposition. Two-dimensional pulse arrays place high hardware demands on near-eye display devices, which, given their limited hardware, can easily lead to inefficient Jacobian matrix decomposition. Furthermore, with a large Jacobian matrix dimension, Givens Rotation-based QR decomposition requires multiple accesses to the memory space caching the Jacobian matrix, increasing memory bandwidth consumption. Both the inefficient Jacobian decomposition and increased memory bandwidth consumption negatively impact the ease of positioning for near-eye display devices. Summary of the Invention

[0004] The main objective of this application is to provide a positioning method, apparatus, device, and storage medium for near-eye display devices, aiming to solve the technical problems of poor efficiency in decomposing the target positioning matrix of near-eye display devices due to hardware limitations, and increased consumption of memory bandwidth due to the need to access the memory space containing the target positioning matrix multiple times, thereby resulting in poor convenience of positioning near-eye display devices.

[0005] In a first aspect, this application provides a positioning method for a near-eye display device, the near-eye display device including a reading unit and at least two iteration units, the positioning method comprising:

[0006] Obtain a preset positioning matrix for a near-eye display device, wherein the preset positioning matrix includes multiple preset column elements;

[0007] Based on the number of iteration units, the reading unit is controlled to read a preset number of preset column elements from the preset positioning matrix; there is a correspondence between the iteration units and the preset column elements.

[0008] According to the preset iteration order corresponding to the at least two iteration units, the iteration units are controlled to perform iteration operations on the corresponding preset column elements to obtain the target column elements;

[0009] Based on the target column elements, determine the target positioning matrix of the near-eye display device;

[0010] The positioning information of the near-eye display device is determined based on the target positioning matrix.

[0011] Secondly, this application provides a positioning device for a near-eye display device, the positioning device comprising:

[0012] A preset matrix acquisition module is used to acquire a preset positioning matrix of a near-eye display device, wherein the preset positioning matrix includes multiple preset column elements;

[0013] The element reading module is used to control the reading unit to read a preset number of preset column elements from the preset positioning matrix according to the number of iteration units; there is a correspondence between the iteration unit and the preset column elements;

[0014] An iteration module is used to control the iteration units to perform iteration operations on the corresponding preset column elements according to a preset iteration order corresponding to at least two iteration units, so as to obtain the target column element;

[0015] The target matrix determination module is used to determine the target positioning matrix of the near-eye display device based on the target column elements;

[0016] The positioning module is used to determine the positioning information of the near-eye display device based on the target positioning matrix.

[0017] Thirdly, this application provides a near-eye display device, which includes a memory and a processor;

[0018] The memory is used to store computer programs;

[0019] The processor is configured to execute the computer program and, in executing the computer program, implement the steps of the near-eye display device positioning method as described above.

[0020] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the positioning method for a near-eye display device as described above.

[0021] This application provides a positioning method, apparatus, device, and storage medium for a near-eye display device. The near-eye display device includes a reading unit and at least two iteration units. The positioning method includes: acquiring a preset positioning matrix of the near-eye display device, the preset positioning matrix including multiple preset column elements; controlling the reading unit to read a preset number of preset column elements from the preset positioning matrix according to the number of iteration units; there is a correspondence between the iteration units and the preset column elements; controlling the iteration units to perform iteration operations on the corresponding preset column elements according to a preset iteration order corresponding to the at least two iteration units to obtain target column elements; determining the target positioning matrix of the near-eye display device based on the target column elements; and determining the positioning information of the near-eye display device based on the target positioning matrix.

[0022] When obtaining the preset positioning matrix of the near-eye display device, which may include a Jacobian matrix, QR decomposition is required to determine the target positioning matrix. If the near-eye display device includes at least two iteration units, these units can be controlled to iterate on the corresponding preset column elements of the preset positioning matrix in parallel according to a preset iteration order, thereby determining the target positioning matrix. Since the iteration units have low hardware requirements for the near-eye display device, iterating on the preset positioning matrix in parallel with at least two units improves the decomposition efficiency. Correspondingly, the reading unit can access the memory space containing the preset positioning matrix in a single access, reading the preset number of preset column elements required by the at least two iteration units, thus reducing memory bandwidth consumption. When the target positioning matrix is ​​determined, it provides positioning information for the near-eye display device, improving the ease of positioning. Attached Figure Description

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 This is a schematic flowchart of a positioning method for a near-eye display device provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of a preset positioning matrix according to an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the iterative operation of an iterative unit on preset column elements according to an embodiment of this application;

[0027] Figure 4 This is a flowchart illustrating the iterative operation of an iterative unit on preset column elements according to an embodiment of this application;

[0028] Figure 5 This is a schematic block diagram of a positioning device for a near-eye display device provided in an embodiment of this application;

[0029] Figure 6 This is a schematic block diagram of a near-eye display device provided in an embodiment of this application. Detailed Implementation

[0030] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0031] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0032] This application provides a positioning method, apparatus, device, and storage medium for a near-eye display device. The positioning method for this near-eye display device can be applied to near-eye display devices. Near-eye display devices may include augmented reality (AR) glasses, mixed reality (MR) glasses, AR helmets, MR helmets, etc., and are not limited thereto.

[0033] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0034] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a positioning method for a near-eye display device according to an embodiment of this application. It should be noted that the positioning method for a near-eye display device provided in this embodiment can be used in a near-eye display device, and is not limited thereto.

[0035] like Figure 1 As shown, the positioning method of the near-eye display device includes steps S101 to S105.

[0036] For example, a near-eye display device includes a reading unit and at least two iteration units.

[0037] For example, the reading unit can read preset column elements included in the preset positioning matrix of the near-eye display device. The iteration unit can perform iterative operations on the preset column elements read by the reading unit. The near-eye display device can, based on the original QR decomposition based on Givens Rotation, set at least two iteration units to perform QR decomposition on the preset positioning matrix of the near-eye display device to determine the positioning information of the near-eye display device.

[0038] S101. Obtain the preset positioning matrix of the near-eye display device. The preset positioning matrix includes multiple preset column elements.

[0039] For example, a near-eye display device can collect sensor data necessary for positioning the device. If the sensor data is sufficient to determine the positioning information of the near-eye display device, a preset positioning matrix can be determined based on this data. Accordingly, based on the matrix properties of the preset positioning matrix, it can include multiple preset columns of elements.

[0040] In some implementations, a preset positioning matrix is ​​determined based on image sensing data and device pose sensing data collected by the near-eye display device.

[0041] For example, a near-eye display device includes an image sensor and an inertial measurement unit (IMU). The near-eye display device can acquire image sensing data of its surrounding environment through the image sensor. The near-eye display device can acquire device pose sensing data through the IMU. This pose sensing data can include the near-eye display device's position and orientation. Since both image sensing data and device pose sensing data can be used to determine the near-eye display device's positioning information, the near-eye display device can combine the image sensing data and device pose sensing data to determine a preset positioning matrix. This preset positioning matrix can be used to determine the target positioning matrix of the near-eye display device. The target positioning matrix can be used to determine the positioning information of the near-eye display device.

[0042] With a predetermined positioning matrix, the positioning information of the near-eye display device can be determined through matrix operations, which will help improve the convenience of positioning the near-eye display device.

[0043] In some implementations, feature extraction and feature matching are performed on the image sensing data to obtain a first feature; the device pose sensing data is integrated to obtain a second feature; and a preset positioning matrix is ​​determined based on the first feature and the second feature.

[0044] For example, near-eye display devices can use visual-inertial odometry (VIO) or visual-inertial system (VINS) to fuse image sensing data and device pose data acquired by the near-eye display device to achieve SLAM.

[0045] like Figure 2 As shown, the near-eye display device can perform feature extraction and feature matching on image sensing data to obtain a first feature. The near-eye display device can also perform integration on device pose sensing data to obtain a second feature. Finally, the near-eye display device can fuse the first and second features to obtain a preset localization matrix. For example, the preset localization matrix may include the H matrix of the state equation. The H matrix can be represented as:

[0046] H = [0 2M×9 H p [H ω1:-1 0 2M×9 H ωN 0 2M×9 ]]

[0047] Where M indicates the dimension of the first feature, 0 2M×9 H p H ω1:-1 and H ωN Used to determine the row and column dimensions in the H matrix, 0 2M×9 Used to indicate the first state matrix corresponding to the H matrix, H p H is used to indicate the second state matrix corresponding to the first feature and the second feature. ω1:-1 H is used to indicate the third state matrix corresponding to rows ω1 to N-1 of matrix H. ωN This is used to indicate the fourth state matrix corresponding to the ωNth row of the H matrix. The row dimension is the dimension of the observation residuals corresponding to all first features, and the column dimension is the dimension of the state variables of all frames within the window corresponding to the first features.

[0048] With a typical value of M of 100×11, that is, the window corresponding to the first feature is 11, and 100 first features can be observed in each window, 0 2M×9 H p H ω1:-1 and H ωN Each of them has a corresponding row dimension of 2200, 02M×9 The corresponding column dimension is 9, H p The corresponding column dimension is 7, H ω1:-1 The first 10 rows in the corresponding window each have a column dimension of 6, H ωN Taking a window with a last row column dimension of 15 as an example, the dimension of the H matrix can be determined as: 2200 × (9 + 7 + 10 × 6 + 9 + 15 + 9) = 2200 × 10⁹. Here, 2200 indicates the row dimension of the H matrix, and 10⁹ indicates the column dimension. Of course, the preset positioning matrix is ​​not limited to the H matrix, and the H matrix is ​​not limited to this; therefore, no restrictions are imposed here.

[0049] Accordingly, since the H matrix has a large dimension, it can be decomposed into a QR matrix to solve a nonlinear optimization problem and obtain the target localization matrix. Based on the target localization matrix, the optimal estimate of the state can be determined. This optimal state estimate can be used to indicate the optimal estimates of the position, orientation, and velocity of the near-eye display device, thus providing the localization information for determining the near-eye display device.

[0050] Thus, by obtaining the preset positioning matrix of the near-eye display device, the near-eye display device can subsequently perform QR decomposition on the preset positioning matrix to determine the corresponding target positioning matrix, thereby determining the positioning information of the near-eye display device.

[0051] S102. Based on the number of iteration units, control the reading unit to read a preset number of preset column elements from the preset positioning matrix; there is a correspondence between the iteration units and the preset column elements.

[0052] For example, when a near-eye display device includes at least two iteration units, the device can perform QR decomposition on a preset positioning matrix in parallel using at least two iteration units, thereby improving the decomposition efficiency of the preset positioning matrix. Based on this, the near-eye display device can control the reading unit to read a preset number of preset column elements from the preset positioning matrix according to the number of iteration units.

[0053] Taking an example where the iteration units include PE0, PE1, and PE2, and the number of iteration units is 3, the near-eye display device performs QR decomposition on the preset positioning matrix in parallel through PE0 to PE2.

[0054] Taking a 9×9 matrix A as an example, the preset positioning matrix can be represented as:

[0055]

[0056] The numerical identifiers of the elements included in each preset column are used to indicate the row and column number of the element in matrix A. For example, element x11 is located in the 1st row and 1st column of matrix A, element x21 is located in the 2nd row and 1st column of matrix A, and so on.

[0057] With three iteration units, the near-eye display device can control the reading unit to read the preset column elements from column 1 to column 3 of matrix A. Correspondingly, the near-eye display device can assign PE0 to the preset column element, PE1 to the preset column element, and PE2 to the preset column element. However, it's not limited to this; the near-eye display device can also first control the reading unit to read the preset column elements from column 4 to column 6 of matrix A, and then control the reading unit to read the preset column elements from column 7 to column 9 of matrix A. Based on this, with three iteration units, the reading unit, for example, needs to access the memory space cached with matrix A three times to complete the reading operation of all preset column elements in matrix A. However, based on the original QR decomposition using Givens Rotation, the reading unit needs to read one column of preset column elements in matrix A each time. Therefore, the reading unit needs to access the memory space cached with matrix A nine times to complete the reading operation of all preset column elements in matrix A.

[0058] Thus, when the reading unit reads a preset number of preset column elements from the preset positioning matrix, it can read these elements in a single access to the memory space containing the preset positioning matrix. This reduces the number of memory accesses and consequently reduces the memory bandwidth consumption of the near-eye display device. Correspondingly, the reading unit can subsequently provide the preset column elements to the corresponding iteration units based on the correspondence between the iteration units and the preset column elements. This allows at least two iteration units to perform parallel iteration operations on their respective preset column elements, thereby improving the efficiency of decomposing the preset positioning matrix.

[0059] In some implementations, the reading unit is controlled to read a preset number of preset column elements from a preset positioning matrix in a left-to-right direction, depending on the number of iteration units.

[0060] Taking a preset positioning matrix A as an example, and the iteration units including PE0 to PE2, the near-eye display device can determine the column number corresponding to each preset column element in matrix A from left to right. For example, the near-eye display device can determine the leftmost preset column element in matrix A as the 1st preset column element, the rightmost preset column element in matrix A as the 9th preset column element, and so on, to determine the order of the preset column elements from the 1st to the 9th preset column elements in matrix A.

[0061] With three iteration units, such as PE0 to PE2, the near-eye display device can control the reading unit to read three preset column elements from matrix A from left to right, such as the first to third preset column elements. The first to third preset column elements read by the reading unit can then be used by PE0 to PE2 to perform iteration operations on their respective first to third preset column elements.

[0062] Of course, this is not the only possibility. After PE0 to PE2 complete their respective iterative operations on the preset elements of columns 1 to 3, the reading unit can continue to read the preset elements of columns 4 to 6 from matrix A in a left-to-right direction, depending on the number of iteration units (e.g., if PE0 to PE2 have 3 iteration units). Similarly, after PE0 to PE2 complete their respective iterative operations on the preset elements of columns 4 to 6, the reading unit can continue to read the preset elements of columns 7 to 9 from matrix A in a left-to-right direction.

[0063] In this way, by controlling the reading unit to read a preset number of preset column elements from the preset positioning matrix from left to right according to the number of iteration units, the near-eye display device can ensure that the reading unit regularly reads the preset column elements in the preset positioning matrix. Subsequent reading units can then provide the corresponding preset column elements to the corresponding iteration units for iterative operations to obtain the target column element. Consequently, when determining the target positioning matrix based on the target column element, the ease of determining the position of the target column element within the target positioning matrix is ​​improved.

[0064] S103. Based on the preset iteration order corresponding to at least two iteration units, control the iteration units to perform iteration operations on the corresponding preset column elements to obtain the target column elements.

[0065] In some implementations, when the control iteration unit performs iteration operations on the corresponding preset column elements, the near-eye display device can control the iteration unit to perform two iteration operations on the preset column elements to obtain the target column element.

[0066] For example, when a near-eye display device includes at least two iteration units, these at least two iteration units have a corresponding preset iteration order. Taking iteration units PE0 to PE2 as an example, the preset iteration order indicates that the iteration order of PE0 precedes the iteration order of PE1, and the iteration order of PE1 precedes the iteration order of PE2. Based on the preset iteration order, the near-eye display device can prioritize controlling PE0 to perform iteration operations on the preset column elements corresponding to PE0, obtaining the target column element corresponding to PE0. After PE0 completes the iteration operation on the preset column elements corresponding to PE0, the near-eye display device can control PE1 to perform iteration operations on the preset column elements corresponding to PE1, obtaining the target column element corresponding to PE1. Similarly, after PE1 completes the iteration operation on the preset column elements corresponding to PE1, the near-eye display device can control PE2 to perform iteration operations on the preset column elements corresponding to PE2, obtaining the target column element corresponding to PE2.

[0067] In some implementations, the iteration unit includes n iteration units, and the preset iteration order indicates that the iteration order of the i-th iteration unit in the n iteration units is before the iteration order of the j-th iteration unit, where j is greater than i; the preset column elements of the preset column number include n preset column elements, and the i-th preset column element in the n preset column elements is located to the left of the j-th preset column element in the n preset column elements; the i-th preset column element corresponds to the i-th iteration unit, and the j-th preset column element corresponds to the j-th iteration unit.

[0068] For example, when the number of iteration units is n, the near-eye display device can control the reading unit to read n preset column elements from a preset positioning matrix from left to right. Accordingly, the near-eye display device can determine that the i-th iteration unit corresponds to the i-th preset column element, and the j-th iteration unit corresponds to the j-th preset column element. Since the preset iteration order indicates that the iteration order of the i-th iteration unit precedes that of the j-th iteration unit, the near-eye display device can first control the i-th iteration unit to perform the iteration operation on the i-th preset column element. After the i-th iteration unit completes the iteration operation on the i-th preset column element, the near-eye display device then controls the j-th iteration unit to perform the iteration operation on the j-th preset column element. This process continues until all preset column elements in the preset positioning matrix have been iterated, obtaining the target column element corresponding to each preset column element.

[0069] Taking a preset positioning matrix including matrix A and iteration units including PE0 to PE2 as an example. Figure 3As shown, the near-eye display device can divide the preset column elements of matrix A into preset column elements from column 1 to column 9 in a left-to-right direction. In the first iteration, the reading unit can provide the preset column elements from column 1 to column 3 read from matrix A to PE0 to PE2 respectively. The near-eye display device can first control PE0 to complete the iteration operation on the preset column element 1, then control PE1 to complete the iteration operation on the preset column element 2, and finally control PE2 to complete the iteration operation on the preset column element 3. In the second iteration, the reading unit can provide the preset column elements from column 4 to column 6 read from matrix A to PE0 to PE2 respectively. The near-eye display device can first control PE0 to complete the iteration operation on the preset column element 4, then control PE1 to complete the iteration operation on the preset column element 5, and finally control PE2 to complete the iteration operation on the preset column element 6. In the third iteration, the reading unit can provide the preset column elements from column 7 to column 9 read from matrix A to PE0 to PE2 respectively. The near-eye display device can first control PE0 to complete the iteration operation on the 7th preset column element, then control PE1 to complete the iteration operation on the 8th preset column element, and finally control PE2 to complete the iteration operation on the 9th preset column element. When PE2 completes the iteration operation on the 9th preset column element, each preset column element in matrix A has been iterated, and the near-eye display device can determine the target column element corresponding to each preset column element in matrix A. The target column element can be used to determine the target positioning matrix of the near-eye display device to determine the positioning information of the near-eye display device.

[0070] Thus, when the near-eye display device controls the iteration units to perform iterative operations on the corresponding preset column elements according to the preset iteration order of at least two iteration units to obtain the target column element, the near-eye display device can control at least two iteration units to perform iterative operations on the preset column elements corresponding to each iteration unit in parallel according to the preset iteration order, which helps to improve the decomposition efficiency of the preset positioning matrix. Correspondingly, since the target column element can be used to determine the target positioning matrix of the near-eye display device to determine the positioning information of the near-eye display device, the determination of the target column element helps to improve the convenience of positioning the near-eye display device.

[0071] For example, a preset column element includes multiple preset row elements. For instance, based on the matrix properties of the preset positioning matrix, for multiple preset column elements included in the preset positioning matrix, the total number of rows corresponding to the preset row elements included in each of the different preset column elements is the same.

[0072] In some implementations, when the control reading unit reads preset column elements, if the total number of rows corresponding to multiple preset row elements is greater than a preset row number threshold, the control reading unit reads preset row elements of the preset column elements including the preset number of rows from the preset positioning matrix in a bottom-up direction; the preset number of rows is less than or equal to the preset row number threshold.

[0073] For example, the preset row number threshold can be determined based on the size of the cache space of the reading unit. When the cache space of the reading unit is limited, it is easy for the reading unit to be unable to directly read all the preset row elements included in the preset column elements when reading the preset number of preset column elements. Therefore, the near-eye display device can control the reading unit to read all the preset row elements included in the preset number of preset column elements multiple times. In some embodiments, the near-eye display device can control the reading unit to read the preset row elements included in the preset column elements from the preset positioning matrix in a bottom-up direction. For example, the near-eye display device can determine the preset row element located at the bottom of the preset column elements as the first preset row element, and the preset row element located at the top of the preset column elements as the last preset row element, and so on, to determine the order of the preset row elements from the first row to the last preset row element.

[0074] like Figure 4 As shown, the near-eye display device controls the reading unit to acquire preset row elements comprising a preset number of preset rows. Taking an iteration unit comprising PE0 to PE2, preset column elements comprising columns 1 to 3, a total of 9 rows corresponding to the preset row elements, and a preset row threshold of 5 as an example. The preset row number can include any number from 1 to 5. The preset row number read by the reading unit each time can be the same or different. For example, the reading unit can sequentially read the 3 preset row elements comprising each of columns 1 to 3 until all preset row elements comprising each of columns 1 to 3 have been read. Another example is that the reading unit can first read the 5 preset row elements comprising each of columns 1 to 3, and then read the 4 preset row elements comprising each of columns 1 to 3. No restrictions are imposed here.

[0075] Accordingly, the near-eye display device can subsequently provide the preset row elements, which include a preset number of rows, read by the reading unit to the corresponding iteration unit so as to determine the target column element corresponding to the preset column element.

[0076] Thus, given the limited size of the cache space of the reading unit, the near-eye display device can flexibly control the reading unit to read preset row elements of a preset number of preset rows, based on the size of the cache space of the reading unit, which helps to improve the flexibility of the reading unit in reading preset column elements.

[0077] In some implementations, the control iteration unit performs iteration operations on a preset number of preset row elements until the iteration operations on all preset row elements are completed, thereby obtaining the target column element.

[0078] like Figure 4 As shown, the near-eye display device controls the iteration unit to perform iteration operations on preset row elements of a preset number of rows. Taking an iteration unit comprising PE0 to PE2, a preset positioning matrix comprising preset column elements from column 1 to column 3, and a preset number of rows of 3 as an example, the near-eye display device can provide the preset row elements from row 1 to row 3 of each of the preset column elements read by the reading unit to PE0 to PE2 respectively. The near-eye display device can first control PE0 to complete the iteration operation on the preset row elements from row 1 to row 3 of the preset column elements, then control PE1 to complete the iteration operation on the preset row elements from row 1 to row 3 of the preset column elements, and finally control PE2 to complete the iteration operation on the preset row elements from row 1 to row 3 of the preset column elements.

[0079] like Figure 4 As shown, the near-eye display device writes the preset row elements of the preset number of rows after the iteration processing is completed back to the reading unit. For example, the near-eye display device can write back the preset row elements of the first column (including rows 1 to 3) after the iteration processing is completed to the reading unit. The near-eye display device can also write back the preset row elements of the second column (including rows 1 to 3) after the iteration processing is completed to the reading unit, and so on.

[0080] like Figure 4As shown, if the iteration unit has not completed the iteration operation on all preset row elements included in the corresponding preset column element, the near-eye display device can cyclically control the reading unit to read the preset row elements of the preset column element, control the iteration unit to perform iteration operations on the preset row elements read by the reading unit, and write the preset row elements of the preset row element after the iteration process is completed back to the reading unit, until the iteration unit completes the iteration operation on all preset row elements included in the preset column element and obtains the target column element. For example, if PE0 to PE2 complete the iteration operation on the preset row elements of the preset number of preset rows included in their respective preset column elements, the near-eye display device can provide the preset row elements of the 4th to 6th rows included in each of the preset column elements of the 1st to 3rd columns read by the reading unit to PE0 to PE2 respectively, and perform iteration operations on the preset row elements of the 4th to 6th rows included in the corresponding preset column element according to the preset iteration order corresponding to PE0 to PE2. This process continues until PE0 completes the iteration operation on all preset row elements included in the first preset column element, obtaining the first target column element corresponding to the first preset column element; PE1 completes the iteration operation on all preset row elements included in the second preset column element, obtaining the second target column element corresponding to the second preset column element; and PE2 completes the iteration operation on all preset row elements included in the third preset column element, obtaining the third target column element corresponding to the third preset column element.

[0081] Thus, when the total number of rows corresponding to the preset row elements is greater than the preset row number threshold, the near-eye display device controls the iteration unit to perform iteration operations on the preset row elements of the preset row number until the iteration operations on all preset row elements are completed and the target column element is obtained. This is beneficial to improve the iteration flexibility of the iteration unit on the preset column elements when the cache space of the reading unit is limited.

[0082] The improved flexibility of the reading unit in reading preset column elements and the improved flexibility of the iteration unit in iterating preset column elements are beneficial to improving the decomposition flexibility of the near-eye display device on the preset positioning matrix, thereby improving the decomposition efficiency of the preset positioning matrix.

[0083] S104. Determine the target positioning matrix of the near-eye display device based on the target column elements.

[0084] For example, when the target column element corresponding to the preset column element is determined, the near-eye display device can determine the position of the target column element in the target positioning matrix.

[0085] In some implementations, the near-eye display device can determine the position of the target column element corresponding to the preset column element in the target positioning matrix based on the position of the preset column element in the preset positioning matrix. Taking matrix A as an example, if the near-eye display device determines the target column elements corresponding to the preset column elements in columns 1 to 9 of matrix A, the position of each target column element in columns 1 to 9 in the target positioning matrix can be determined from left to right, thereby determining the target positioning matrix corresponding to matrix A.

[0086] In some implementations, since the target column elements are obtained by the near-eye display device controlling the iteration units to perform iterative operations on the corresponding preset column elements according to a preset iteration order corresponding to at least two iteration units, the multiple target column elements can have a corresponding definite order. Taking a preset positioning matrix including matrix A as an example: the near-eye display device determines the order of the first target column elements corresponding to the first preset column element earlier than the order of the second target column elements corresponding to the second preset column element. Correspondingly, the near-eye display device determines the order of the target column elements corresponding to the second preset column element earlier than the order of the third target column element corresponding to the third preset column element. And so on, the determination order of the target column elements corresponding to each of the first to ninth preset column elements in matrix A can be determined. Accordingly, the near-eye display device can determine the position of each of the first to ninth target column elements in the target positioning matrix from left to right according to the determination order of the target column elements corresponding to each of the first to ninth preset column elements, thereby determining the target positioning matrix corresponding to matrix A.

[0087] In this way, the near-eye display device determines the target positioning matrix based on the target column elements. The target positioning matrix can be used to determine the positioning information of the near-eye display device, which helps to improve the convenience of positioning the near-eye display device.

[0088] S105. Determine the positioning information of the near-eye display device based on the target positioning matrix.

[0089] For example, with a defined target positioning matrix, near-eye display devices can simplify the computational complexity of determining their positioning information. Furthermore, the availability of the target positioning matrix for determining the positioning information of near-eye display devices improves the ease of positioning them.

[0090] The near-eye display device positioning method provided in the above embodiments includes: obtaining a preset positioning matrix of the near-eye display device, the preset positioning matrix including multiple preset column elements; controlling a reading unit to read a preset number of preset column elements from the preset positioning matrix according to the number of iteration units; there is a correspondence between the iteration units and the preset column elements; controlling the iteration units to perform iteration operations on the corresponding preset column elements according to a preset iteration order corresponding to at least two iteration units to obtain target column elements; determining the target positioning matrix of the near-eye display device according to the target column elements; and determining the positioning information of the near-eye display device according to the target positioning matrix.

[0091] When obtaining the preset positioning matrix of the near-eye display device, which may include a Jacobian matrix, QR decomposition is required to determine the target positioning matrix. If the near-eye display device includes at least two iteration units, these units can be controlled to iterate on the corresponding preset column elements of the preset positioning matrix in parallel according to a preset iteration order, thereby determining the target positioning matrix. Since the iteration units have low hardware requirements for the near-eye display device, iterating on the preset positioning matrix in parallel with at least two units improves the decomposition efficiency. Correspondingly, the reading unit can access the memory space containing the preset positioning matrix in a single access, reading the preset number of preset column elements required by the at least two iteration units, thus reducing memory bandwidth consumption. When the target positioning matrix is ​​determined, it provides positioning information for the near-eye display device, improving the ease of positioning.

[0092] Please see Figure 5 , Figure 5 This is a schematic block diagram of a positioning device for a near-eye display device according to an embodiment of this application. The positioning device can be configured in a near-eye display device to perform the aforementioned positioning method for the near-eye display device. Near-eye display devices may include AR glasses, VR glasses, AR helmets, VR helmets, etc., and are not limited thereto.

[0093] like Figure 5 As shown, the positioning device includes a preset matrix acquisition module 110, an element reading module 120, an iteration module 130, a target matrix determination module 140, and a positioning module 150.

[0094] The preset matrix acquisition module 110 is used to acquire the preset positioning matrix of the near-eye display device, wherein the preset positioning matrix includes multiple preset column elements.

[0095] The element reading module 120 is used to control the reading unit to read a preset number of preset column elements from the preset positioning matrix according to the number of iteration units; the iteration unit and the preset column elements have a corresponding relationship.

[0096] The iteration module 130 is used to control the iteration unit to perform iteration operations on the corresponding preset column elements according to the preset iteration order corresponding to at least two iteration units, so as to obtain the target column element.

[0097] The target matrix determination module 140 is used to determine the target positioning matrix of the near-eye display device based on the target column elements.

[0098] The positioning module 150 is used to determine the positioning information of the near-eye display device based on the target positioning matrix.

[0099] For example, the element reading module 120 includes a first element reading submodule.

[0100] The first element reading submodule is used to control the reading unit to read a preset number of preset column elements from the preset positioning matrix in a left-to-right direction, based on the number of iteration units.

[0101] For example, iteration module 130 includes a first iteration submodule.

[0102] The first iteration submodule is configured such that the iteration unit comprises n iteration units, and the preset iteration order indicates that the iteration order of the i-th iteration unit among the n iteration units precedes the iteration order of the j-th iteration unit, where j is greater than i; the preset column elements of the preset column number comprise n preset column elements, and the i-th preset column element among the n preset column elements is located to the left of the j-th preset column element among the n preset column elements; the i-th preset column element corresponds to the i-th iteration unit, and the j-th preset column element corresponds to the j-th iteration unit.

[0103] For example, the preset column element includes multiple preset row elements.

[0104] The first element reading submodule includes the second element reading submodule.

[0105] The second element reading submodule is used to control the reading unit to read the preset row elements of the preset column elements from the preset positioning matrix in a bottom-up direction if the total number of rows corresponding to the preset row elements is greater than the preset row number threshold when controlling the reading unit to read the preset column elements; the preset row number is less than or equal to the preset row number threshold.

[0106] For example, the first iteration submodule includes the second iteration submodule.

[0107] The second iteration submodule is used to control the iteration unit to perform iteration operations on the preset row elements of the preset number of rows until the iteration operations on all the preset row elements are completed to obtain the target column elements.

[0108] For example, the preset matrix acquisition module 110 includes a preset matrix determination submodule.

[0109] The preset matrix determination submodule is used to determine the preset positioning matrix based on the image sensing data and device pose sensing data collected by the near-eye display device.

[0110] For example, the preset matrix determination submodule includes a first feature determination submodule, a second feature determination submodule, and a feature fusion submodule.

[0111] The first feature determination submodule is used to perform feature extraction and feature matching processing on the image sensing data to obtain the first feature.

[0112] The second feature determination submodule is used to perform integral processing on the device pose sensing data to obtain the second feature.

[0113] The feature fusion submodule is used to determine the preset positioning matrix based on the first feature and the second feature.

[0114] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and its modules and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0115] The method of this application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0116] For example, the above-described method and apparatus can be implemented as a computer program that can run on a near-eye display device to locate the near-eye display device. For example, the near-eye display device may include VR glasses, AR glasses, VR headsets, AR headsets, etc., and is not limited thereto.

[0117] Please see Figure 6 , Figure 6 This is a schematic block diagram of the structure of a near-eye display device provided in an embodiment of this application.

[0118] like Figure 6 As shown, the near-eye display device includes a memory and a processor. The memory and processor can be connected via a system bus, and the memory may include a storage medium and internal memory.

[0119] The storage medium can store the operating system and computer programs. When the computer program is executed, it enables the processor to perform any positioning method for the near-eye display device.

[0120] The processor provides computing and control capabilities to support the operation of the entire near-eye display device.

[0121] The internal memory provides an environment for the execution of computer programs stored in the storage medium. When the computer program is executed by the processor, it enables the processor to perform any positioning method for near-eye display devices.

[0122] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the near-eye display device to which the present application is applied. A specific near-eye display device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0123] It should be understood that a processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other convertible logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0124] In one embodiment, the processor is configured to execute a computer program and, when executing the computer program, perform the following steps:

[0125] Obtain a preset positioning matrix for a near-eye display device, wherein the preset positioning matrix includes multiple preset column elements;

[0126] Based on the number of iteration units, the reading unit is controlled to read a preset number of preset column elements from the preset positioning matrix; there is a correspondence between the iteration units and the preset column elements.

[0127] According to the preset iteration order corresponding to the at least two iteration units, the iteration units are controlled to perform iteration operations on the corresponding preset column elements to obtain the target column elements;

[0128] Based on the target column elements, determine the target positioning matrix of the near-eye display device;

[0129] The positioning information of the near-eye display device is determined based on the target positioning matrix.

[0130] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of positioning the near-eye display device described above can be referred to the corresponding process in the aforementioned embodiments of the positioning method for the near-eye display device, and will not be repeated here.

[0131] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method implemented can be referred to in various embodiments of the positioning method of the near-eye display device of this application.

[0132] The computer-readable storage medium can be an internal storage unit of the near-eye display device described in the foregoing embodiments, such as the hard disk or memory of the near-eye display device. Alternatively, the computer-readable storage medium can be an external storage device of the near-eye display device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the near-eye display device.

[0133] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0134] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0135] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments. The above description is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A positioning method for a near-eye display device, characterized in that, The near-eye display device includes a reading unit and at least two iteration units; The positioning method includes: Obtain a preset positioning matrix for a near-eye display device, wherein the preset positioning matrix includes multiple preset column elements; Based on the number of iteration units, the reading unit is controlled to read a preset number of preset column elements from the preset positioning matrix; there is a correspondence between the iteration units and the preset column elements. According to the preset iteration order corresponding to the at least two iteration units, the iteration units are controlled to perform iteration operations on the corresponding preset column elements to obtain the target column elements; Based on the target column elements, determine the target positioning matrix of the near-eye display device; The positioning information of the near-eye display device is determined based on the target positioning matrix. The step of controlling the reading unit to read a preset number of preset column elements from the preset positioning matrix based on the number of iteration units includes: Based on the number of iteration units, the reading unit is controlled to read a preset number of preset column elements from the preset positioning matrix in a left-to-right direction; The iterative unit includes n iterative units, and the preset iterative order indicates that the iteration order of the i-th iterative unit in the n iterative units is before the iteration order of the j-th iterative unit, where j is greater than i; the preset column elements of the preset column number include n preset column elements, and the i-th preset column element in the n preset column elements is located to the left of the j-th preset column element in the n preset column elements; the i-th preset column element corresponds to the i-th iterative unit, and the j-th preset column element corresponds to the j-th iterative unit.

2. The positioning method according to claim 1, characterized in that, The preset column elements include multiple preset row elements; The control of the reading unit to read a preset number of preset column elements from the preset positioning matrix in a left-to-right direction includes: When controlling the reading unit to read the preset column elements, if the total number of rows corresponding to the multiple preset row elements is greater than the preset row number threshold, the reading unit is controlled to read the preset row elements of the preset column elements from the preset positioning matrix in a bottom-to-top direction, including the preset number of preset row elements. The preset number of rows is less than or equal to the preset number of rows threshold.

3. The positioning method according to claim 2, characterized in that, The control of the iteration unit to perform iteration operations on the corresponding preset column elements to obtain the target column elements includes: The iteration unit is controlled to perform iteration operations on the preset row elements of a preset number of rows until the iteration operations on all preset row elements are completed, thereby obtaining the target column element.

4. The positioning method according to any one of claims 1 to 3, characterized in that, The process of obtaining the preset positioning matrix of the near-eye display device includes: The preset positioning matrix is ​​determined based on the image sensing data and device pose sensing data collected by the near-eye display device.

5. The positioning method according to claim 4, characterized in that, The step of determining the preset positioning matrix based on the image sensing data and device pose sensing data collected by the near-eye display device includes: The image sensing data is subjected to feature extraction and feature matching to obtain the first feature; The device pose sensing data is integrated to obtain the second feature; The preset positioning matrix is ​​determined based on the first feature and the second feature.

6. A positioning device for a near-eye display device, characterized in that, The positioning device includes: A preset matrix acquisition module is used to acquire a preset positioning matrix of a near-eye display device, wherein the preset positioning matrix includes multiple preset column elements; The element reading module is used to control the reading unit to read a preset number of preset column elements from the preset positioning matrix according to the number of iteration units; there is a correspondence between the iteration unit and the preset column elements; An iteration module is used to control the iteration units to perform iteration operations on the corresponding preset column elements according to a preset iteration order corresponding to at least two iteration units, so as to obtain the target column element; The target matrix determination module is used to determine the target positioning matrix of the near-eye display device based on the target column elements; A positioning module is used to determine the positioning information of the near-eye display device based on the target positioning matrix; The step of controlling the reading unit to read a preset number of preset column elements from the preset positioning matrix based on the number of iteration units includes: Based on the number of iteration units, the reading unit is controlled to read a preset number of preset column elements from the preset positioning matrix in a left-to-right direction; The iterative unit includes n iterative units, and the preset iterative order indicates that the iteration order of the i-th iterative unit in the n iterative units is before the iteration order of the j-th iterative unit, where j is greater than i; the preset column elements of the preset column number include n preset column elements, and the i-th preset column element in the n preset column elements is located to the left of the j-th preset column element in the n preset column elements; the i-th preset column element corresponds to the i-th iterative unit, and the j-th preset column element corresponds to the j-th iterative unit.

7. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the steps of the positioning method for the near-eye display device as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the positioning method for the near-eye display device as described in any one of claims 1 to 5.

Images