Method for accelerating simultaneous localization and mapping and device using the same

Abstract

An electronic device performing a simultaneous localization and mapping (SLAM) operation is provided, including: an image database; a factor graph database; and a processor configured to: receive a map image in which a position of the electronic device is localized from the image database, to perform relocalization of the position of the electronic device in the map image, to receive data for calculating components of the electronic device related to six DoF from the factor graph database, to construct a matrix including the components related to the six DoF based on the received data, and to load and rearrange the matrix and a vector selected from a plurality of vectors stored in a memory to sequentially obtain first data and second data by performing a calculation on each block of each row of the matrix and the vector, and to shift the second data to a storage location of the first data.

Description

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to Korean Patent Application No. 10-2020-0148128, filed with the Korean Intellectual Property Office on November 6, 2020, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The apparatus and methods consistent with the example embodiments relate to a method for accelerating matrix operations to shorten the processing time for performing Simultaneous Localization and Mapping (SLAM), and an apparatus using the method. Background Technology

[0004] To construct a map in virtual space, processing is required to analyze images received from cameras, etc., and to determine coordinates. Virtual spaces represented by augmented reality (AR) and virtual reality (VR) can be achieved using portable devices such as glasses (e.g., AR glasses).

[0005] Simultaneous Localization and Mapping (SLAM) can refer to a technology used to construct maps of virtual spaces in real time. Various devices can use SLAM to update maps of virtual spaces in real time, and may need to perform rapid calculations in response to the movement of users using the devices. Summary of the Invention

[0006] SLAM can be implemented using a front-end processor and a back-end processor. The front-end processor receives sensor information and performs feature extraction and spatial coordinate calculation, while the back-end processor optimizes map and current location information based on the front-end output. The front-end processor can be primarily implemented using visual odometry based on camera information and an inertial navigation system (INS) using an inertial measurement unit (IMU). Various sensor fusion schemes can be applied depending on the type of front-end processor. When optimizing the location information generated from sensor data along with map data at the back-end, the amount of mathematical computation can increase significantly depending on the size of the map and the size and accuracy of the sensor data.

[0007] Typically, SLAM can be implemented using a combination of multiple cameras and IMUs, and the mathematical operations used for SLAM can be performed by processors with varying performance specifications on different platforms. Specifically, optimizing the backend processor requires a large amount of mathematical computation and can significantly impact the overall performance of SLAM.

[0008] SLAM to be integrated with virtual space can be configured for low power consumption to improve the wearability of SLAM-enabled devices. With low-power, low-performance backend processors, mathematical operations can take a long time to process. Additionally, in the case of tethered SLAM used to connect to mobile devices, problems may arise due to the weight of the wires used for the connection between devices or data latency. Devices with embedded high-performance processors (e.g., AR glasses) may increase their size and power consumption.

[0009] According to one aspect of an example embodiment, an electronic device is provided that performs simultaneous localization and mapping (SLAM) operations. The electronic device includes: an image database; a factor graph database; and a processor configured to: receive from the image database a map image in which the location of the electronic device is located, to perform relocation of the electronic device's location in the map image; receive from the factor graph database data for calculating components of the electronic device related to six degrees of freedom (DoF); construct a matrix including components related to the six DoF based on the received data; and load and rearrange the matrix and vectors selected from a plurality of vectors stored in memory to sequentially obtain first data and second data by performing calculations on each block of each row of the matrix and the vectors, and shift the second data to the storage location of the first data.

[0010] The processor can also be configured to reuse as many elements as the block size in the calculations of the matrix and the vector.

[0011] The processor can also be configured to compute only the diagonal elements of the scattering matrix by using properties of the scattering matrix derived from the map image and the data used to compute components associated with the six DoFs.

[0012] The processor can also be configured to compute, in the computation performed on each block of each row of the constructed matrix, the same number of transpose elements as the maximum size of each row block.

[0013] The processor can also be configured to output the shifted second data and accumulate the second data and the first data.

[0014] The processor can also be configured to simultaneously perform the accumulation of the first data and the second data and the shifting of the second data.

[0015] The processor can also be configured to perform the shifting of the second data by using a shift register.

[0016] The elements of the constructed matrix may include matrix position components, rotational components, linear velocity components, acceleration components, and angular velocity components relative to the three axes of the electronic device.

[0017] The processor can also be configured to control the interval between the first output time of the first data and the second output time of the second data according to a preset time period.

[0018] The image database can store map images generated based on images received from the camera of the electronic device.

[0019] According to one aspect of an example embodiment, a method for accelerating simultaneous localization and mapping (SLAM) performed by an electronic device is provided, the method comprising: receiving a map image from an image database in which the location of the electronic device is located, to perform relocation of the electronic device's location in the map image; receiving data from a factor graph database of the electronic device for calculating components of the electronic device related to six degrees of freedom (DoF); constructing a matrix including components related to the six DoF based on the received data; and loading and rearranging the matrix and vectors selected from a plurality of vectors stored in memory to sequentially obtain first data and second data by performing calculations on each block of each row of the matrix and the vectors, and shifting the second data to the storage location of the first data.

[0020] The method may further include reusing an equal number of elements as the size of the block in the computation performed on the matrix and the vector.

[0021] The method may further include: calculating only the diagonal elements of the scattering matrix by using properties of the scattering matrix, which is derived from the map image and the data used to calculate components associated with the six DoFs.

[0022] The method may further include: in the computation performed on each block of each row of the constructed matrix, computing as many transpose elements as the maximum size of each row block.

[0023] The method may further include: outputting the shifted second data, and accumulating the second data and the first data.

[0024] The output of the second data may include simultaneously performing the accumulation of the first data and the second data and the shifting of the second data.

[0025] The method may further include: performing the shifting of the second data by using a shift register.

[0026] The construction of the matrix may include calculating the position component, rotation component, linear velocity component, acceleration component, and angular velocity component of the matrix relative to the three axes of the electronic device.

[0027] The method may further include: controlling the interval between the first output time of the first data and the second output time of the second data according to a preset time period.

[0028] The method may further include storing the map data generated based on images received from the camera of the electronic device in the image database. Attached Figure Description

[0029] The above and / or other aspects, features, and advantages will become more apparent from the accompanying drawings, which describe certain exemplary embodiments, in which:

[0030] Figure 1 This is a block diagram illustrating the components of a preconditioned conjugate gradient (PCG) solver according to various embodiments;

[0031] Figure 2 This is an example diagram illustrating the configuration of a PCG solver according to various embodiments;

[0032] Figure 3 This is an example diagram illustrating a matrix-vector selector according to various embodiments;

[0033] Figure 4 This is an example diagram illustrating a matrix-vector multiplier according to various embodiments;

[0034] Figure 5 This is an example diagram illustrating a matrix-vector accumulator according to various embodiments;

[0035] Figure 6 This is a flowchart of data operations performed by the PCG solver and their outputs according to various embodiments;

[0036] Figure 7 This is a flowchart of mathematical operations performed by a PCG solver according to various embodiments;

[0037] Figure 8 These are example diagrams relating to mathematical operations performed by the PCG solver and their output, according to various embodiments; and

[0038] Figure 9 These are example diagrams relating to mathematical simplification and result output of the PCG solver according to various embodiments. Detailed Implementation

[0039] The exemplary embodiments are described in more detail below with reference to the accompanying drawings.

[0040] In the following description, the same reference numerals are used for the same elements, even in different drawings. Matters defined in the description, such as detailed constructions and elements, are provided to aid in a comprehensive understanding of the exemplary embodiments. However, it will be apparent that the exemplary embodiments can be implemented without those specifically defined matters. Furthermore, well-known functions or constructions are not described in detail, as they would be obscured by unnecessary detail.

[0041] When used herein, the term “and / or” includes any and all combinations of one or more of the associated enumerations. Expressions such as “at least one of…” modify the entire list of elements when following a list of elements and do not modify individual elements of the list. For example, the expression “at least one of a, b, and c” should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or any variation of the foregoing examples.

[0042] The terminology used in the embodiments is selected from commonly used terms that are currently widely used; however, these terms may differ depending on the intent of those skilled in the art, precedents, or the emergence of new technologies. Additionally, in certain cases, the terms are selected at the discretion of the applicant of this disclosure, in which case the meaning of those terms will be described in detail in the corresponding sections of the detailed description. Therefore, the terminology used in this disclosure is not merely a designation of terms, but is defined based on the meaning of the terms and content throughout this disclosure.

[0043] Terms such as “comprising” or “including” as used herein should not be construed as necessarily including all the various elements or operations described herein, and it should be understood that some of the elements or operations may be omitted, or additional elements or operations may be provided.

[0044] In the embodiments, it will be understood that when an element is referred to as being “connected to” another element, the element may be directly connected to the other element, or may be electrically connected to the other element while having an intermediate element in between.

[0045] It will be understood that although terms such as “first” or “second” may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another.

[0046] Figure 1 This is a block diagram illustrating the elements of a preconditional conjugate gradient (PCG) solver according to various embodiments.

[0047] refer to Figure 1The PCG solver 100 may include a back-end processor 110, an image database 120, and a factor graph database 130. For example, the PCG solver 100 can perform complex matrix multiplication operations. The PCG solver 100 can perform repeatedly performed matrix and / or vector multiplication operations. For example, the PCG solver 100 can accelerate SLAM execution. The PCG solver 100 can efficiently handle matrix and / or vector multiplication operations, thereby enabling the back-end processor to execute SLAM quickly. (Reference) Figure 1 The components of the PCG solver 100 are merely exemplary and not limited thereto, and some of the components may be replaced and / or additional components may be further included.

[0048] Simultaneous Localization and Mapping (SLAM) is a technique for performing localization and map construction simultaneously. For example, various electronic devices can perform SLAM to estimate their position and construct a map of their environment in real time. SLAM can include visual SLAM. For example, visual SLAM can include a visual odometry system. A visual odometry system can calculate and accumulate the relative position of the electronic device for each frame. For example, a visual odometry system can calculate the position of the electronic device in real time from the starting point where the electronic device begins to move. Because visual odometry systems perform calculations based on images received from a camera, errors may occur due to noise in the received images. In addition to visual odometry systems, SLAM can also utilize loop closure and graph optimization. For example, when the electronic device returns to the starting point, its trajectory can be corrected by using a loop closure method.

[0049] refer to Figure 1The back-end processor 110 can perform mathematical operations to optimize the SLAM performed by the PCG solver 100. For example, the electronic device may include a front-end processor configured to receive sensor data derived from multiple sensors (e.g., cameras, light detection and ranging (LiDAR) devices, etc.) and perform sensor fusion to combine the sensor data obtained from the multiple sensors. The back-end processor 110 can receive the results of sensor fusion and calculate the accumulated travel distance based on the sensor fusion performed by the front-end processor. The back-end processor 110 can receive data from the front-end processor and repeatedly perform mathematical operations. For example, the back-end processor 110 can perform mathematical operations on matrices and / or vectors to estimate the position of the electronic device and construct a map. The back-end processor 110 can estimate the position of the electronic device on the constructed map. For example, the back-end processor 110 can estimate the position of an electronic device moving on the constructed map based on repeatedly performed mathematical operations. The back-end processor 110 can estimate the position of the electronic device in real time and can update data related to the estimated position in real time. The front-end processor and back-end processor 110 can be integrated into a single processor or implemented using separate processors.

[0050] Image database 120 can store images received from an image signal processor (ISP). Image database 120 can send image data for performing repositioning. For example, backend processor 110 can receive image data needed to perform SLAM from image database 120. Image database 120 can receive images (e.g., map images showing an area including the location of an electronic device) from the image signal processor in real time and send image data needed to estimate the location of the electronic device in real time to backend processor 110. For example, when the electronic device moves, backend processor 110 can receive image data from image database 120 and then perform repositioning on the electronic device.

[0051] The factor graph database 130 can store data received from an inertial measurement unit (IMU). The IMU can be implemented using any one or any combination of accelerometers, gyroscopes, magnetometers, and Global Positioning System (GPS) sensors, and can measure the position, angular velocity, and orientation of an object. For example, the factor graph database 130 can store data related to three-axis coordinates measured by the inertial measurement unit. The factor graph database 130 can send data related to translational and rotational movements of the electronic device, relating to its three-axis motion. For example, the back-end processor 110 can receive data from the factor graph database 130 for calculating components related to the six degrees of freedom (6DoF) of the electronic device.

[0052] The electronic device embedded with the PCG solver 100 may include inertial sensors, accelerometers, angular velocity sensors, magnetic sensors, geomagnetic sensors, etc. An electronic device capable of detecting six degrees of freedom can be used where the accelerometer and angular velocity sensors operate along three mutually orthogonal axes. For example, the back-end processor 110 can receive the detection outputs from the electronic device's accelerometer and angular velocity sensors and then calculate the relative position of the electronic device based on the linear acceleration and angular velocity relative to the three axes. The back-end processor 110 can calculate the rotation angle of the electronic device's reference coordinate system based on the angular velocity, as well as calculate the velocity as the integral of acceleration and the position as the integral of velocity.

[0053] Figure 2 This is an example diagram illustrating the configuration of a PCG solver 100 according to various embodiments.

[0054] refer to Figure 2 The PCG solver 200 may include blocks that perform various functions. For example, the PCG solver 200 may include a state controller, a memory array, a preprocessing unit, a multiplier unit, and a postprocessing unit. Figure 2 The configuration of the PCG solver 200 shown is merely exemplary and not limited thereto, and some blocks may be omitted or replaced, and / or additional blocks may be further included. The PCG solver 200 may include... Figure 1 At least one of the configurations and / or functions of the PCG solver 100 shown.

[0055] refer to Figure 2 The preprocessing unit can select matrices and / or vectors from the memory array based on the matrices input to the PCG solver 200. For example, the matrix-vector selector 210 of the preprocessing unit can select matrices and / or vectors to be multiplied by the input matrix. In addition to the matrix-vector selector 210, a vector-scalar selector can select vectors and / or scalars. For example, the vector-scalar selector can select vectors and / or scalars to be multiplied by the input matrix.

[0056] refer to Figure 2The matrix-vector multiplier 220 can perform the multiplication of the matrix input to the PCG solver 200 with the matrix and / or vector selected by the matrix-vector selector 210. For example, the matrix-vector multiplier 220 can perform the multiplication of the input matrix with the matrix selected by the matrix-vector selector 210. The matrix-vector multiplier 220 can also perform the multiplication of the input matrix with the vector selected by the matrix-vector selector 210. For example, the matrix-vector multiplier 220 can perform the multiplication of the input matrix with the vector selected by the vector-scalar selector. The vector-scalar multiplier 240 can also perform the multiplication of the input matrix with the scalar selected by the vector-scalar selector.

[0057] refer to Figure 2 The post-processing unit of the PCG solver 200 may include a matrix-vector accumulator 230. For example, the matrix-vector accumulator 230 may accumulate at least a portion of the result of the multiplication operation performed by the matrix-vector multiplier 220 as output. The matrix-vector accumulator 230 may store at least a portion of the output in the memory of the PCG solver 200 or an electronic device embedded with the PCG solver 200. A scalar accumulator may accumulate at least a portion of the result of the multiplication operation performed by the vector-scalar multiplier 240 as output. A vector adder may add at least some of the vectors in the result of the multiplication operation performed by the vector-scalar multiplier 240. A pre-conditional vector adder may perform addition on at least some of the vectors according to predefined conditions.

[0058] refer to Figure 2 The processing results of the post-processing unit of the PCG solver 200 can be returned to the memory array. For example, an electronic device can estimate its position in real time by performing SLAM, and the output data already used for position estimation can be reused as new input to update the position in real time.

[0059] Figure 3 This is an example diagram illustrating a matrix-vector selector 210 according to various embodiments.

[0060] refer to Figure 3 The matrix-vector selector 210 can select a matrix by using matrix data S as input. For example, the matrix data S can be used to compute the Hessian matrix using the Schur complement. The matrix data S can be stored in the matrix data register 210a.

[0061] refer to Figure 3The matrix-vector selector 210 can select a vector by using vector data b as input. Vector data b can be input and selected in matrix form. For example, vector data b can be stored in vector register 210b, used for transposing matrices, via vector registers (e.g., vector register 1, vector register 2, vector register 3, and vector register 4). Vector data b can also be stored in vector register 210c, used for the original matrix, via vector registers.

[0062] refer to Figure 3 The matrix-vector selector 210 can select matrices and / or vectors for optimizing the SLAM operations of the PCG solver 200. Optimization of the SLAM operations may mean acceleration of the SLAM operations. For example, the speed of the SLAM operations can depend on the processing speed of matrix and / or vector multiplication operations. To improve the processing speed of matrix and / or vector multiplication operations, preprocessing can be performed on the transpose elements of each row of the input matrix. The back-end processor 110 of the PCG solver 200 can perform matrix and / or vector selection using the matrix-vector selector 210 while performing preprocessing to improve the processing speed of multiplication operations.

[0063] Figure 4 This is an example diagram illustrating a matrix-vector multiplier 220 according to various embodiments.

[0064] refer to Figure 4 The matrix-vector multiplier 220 can perform matrix and / or vector multiplication operations. For example, the matrix-vector multiplier 220 can perform a multiplication operation between a matrix (e.g., a scattering matrix) stored in the matrix data register 210a and a vector stored in the vector register 210c for the original matrix. The scattering matrix (S matrix) may correspond to a symmetric positive definite (SPD) matrix. The matrix-vector multiplier 220 can perform a multiplication operation between a matrix (e.g., an S matrix) stored in the matrix data register 210a and a vector stored in the vector register 210b for the transpose matrix. The S matrix may contain camera-related (three-axis) position, camera-related (three-axis) rotation, IMU-related (three-axis) linear velocity, IMU-related (three-axis) acceleration, and IMU-related (three-axis) angular velocity as its elements. The PCG solver (e.g., Figure 1 PCG solver 100 or Figure 2 The back-end processor of the PCG solver 200 (e.g., Figure 1 The backend processor 110 can use images from a database (e.g., Figure 1 Image database 120) and factor graph database (e.g., Figure 1 The S matrix is ​​configured using data received from the factor graph database 130.

[0065] refer to Figure 4 The matrix-vector multiplier 220 may include: three sets of eighteen multipliers each, three rounding and clipping units, and three adders. The matrix-vector multiplier 220 may further include a data rearranger, eighteen sets of three multipliers each, eighteen rounding and clipping units, and eighteen adders. For example, the number of multipliers and sets of multipliers in the matrix-vector multiplier 220 may depend on the size of the matrix and / or vector to which the multiplication operation is performed. The size of the matrix and / or vector can be represented as "n*n".

[0066] When performing multiplication relative to a vector stored in the vector register 210c used for the original matrix, the matrix-vector multiplier 220 can obtain a product of size n*1 as the result of multiplying an n*n matrix with a vector of size n*1. For example, the matrix-vector multiplier 220 can perform eighteen multiplication operations and then perform three significant addition operations. A significant addition can mean that the result of the multiplication operation has not converged to zero.

[0067] refer to Figure 4 The results of multiplication operations performed by the matrix-vector multiplier 220 can be stored in the original matrix register or the transpose matrix register. For example, the result of vector multiplication of the original matrix can be stored in the original matrix register. The result of vector multiplication of the transpose matrix can be stored in the transpose matrix register. Figure 3 and Figure 4 The matrix registers can be the same. For example, Figure 3 The matrix register can be used with Figure 4 The original matrix registers are the same, and Figure 4 The original matrix register can store and Figure 3 The result of multiplication operations related to the data in the matrix register. Figure 3 The matrix register can be used with Figure 4 The transpose matrix registers are the same, and Figure 4 The transpose matrix register can store and Figure 3 The result of multiplication operations related to the data in the matrix register.

[0068] The number of multipliers, rounding and clipping units, adders, and data rearrangers in the matrix-vector multiplier 220 can be limited to... Figure 4 The quantities shown. For example, a single operational component of the matrix-vector multiplier 220 can be determined based on the size of the matrix and / or vector to which the multiplication operation is performed.

[0069] Figure 5 This is an example diagram illustrating a matrix-vector accumulator 230 according to various embodiments.

[0070] refer to Figure 5 The matrix-vector accumulator 230 can accumulate at least a portion of the result of the multiplication operation performed by the matrix-vector multiplier 220 as output. For example, the matrix-vector accumulator 230 can accumulate and store the data of the original matrix register and the data of the transposed matrix register in the order of output. The matrix-vector accumulator 230 can sequentially accumulate the results output by matrix and / or vector multiplication operations.

[0071] refer to Figure 5 The matrix-vector accumulator 230 can separate the data in the transposed matrix register by using a data separator. For example, the data separator can separate the data of the product of multiplication operations stored in the transposed matrix register, so that each row is defined as a block.

[0072] refer to Figure 5 The matrix-vector accumulator 230 can reuse data from the original matrix register and the transposed matrix register. For example, the matrix-vector accumulator 230 can reuse the accumulated and stored results to estimate the location of an electronic device embedded with the PCG solver 200.

[0073] Figure 6 It is a flowchart of data operations performed by the PCG solver 200 and their output according to various embodiments.

[0074] refer to Figure 6 In operation 610, the PCG solver (e.g., Figure 1 PCG solver 100 or Figure 2 The PCG solver 200 can obtain data from image databases (e.g., Figure 1 The image database 120 receives images. For example, the image being received may have already been stored in the image database after being received from the image signal processor. The back-end processor of the PCG solver (e.g., Figure 1 The back-end processor 110 can perform mathematical operations for SLAM on images stored in the image database.

[0075] refer to Figure 6 In operation 620, the PCG solver can draw from a factor graph database (e.g., Figure 1The factor graph database 130 receives data. For example, the factor graph database can store data used to extract components related to the six degrees of freedom by using an IMU based on the motion of an electronic device. The back-end processor of the PCG solver can perform mathematical operations for SLAM on the data stored in the factor graph database.

[0076] refer to Figure 6 In operation 630, the PCG solver can perform calculations of components related to the six degrees of freedom to estimate the position of the electronic device. For example, the PCG solver can perform calculations of components related to the six degrees of freedom using images and data received from an image database and a factor graph database. The six-degree-of-freedom components used to estimate the position of the electronic device can be associated with translational movements (e.g., forward / backward movement on the x-axis, left / right movement on the y-axis, and up / down movement on the z-axis) and rotational movements (e.g., yaw, pitch, and roll) relative to the three axes of the electronic device (e.g., x-axis, y-axis, and z-axis). The back-end processor of the PCG solver can perform calculations of the six-degree-of-freedom components using an IMU consisting of an accelerometer, angular velocity sensor, etc., and a camera of the electronic device. For example, the components related to the six degrees of freedom can include position, rotation, linear velocity, acceleration, and angular velocity relative to the three axes.

[0077] refer to Figure 6 In Operation 640, the PCG solver can configure the matrix based on factors. For example, the PCG solver can configure the matrix based on fifteen factors using a back-end processor. These fifteen factors can be obtained by calculating the components associated with the six degrees of freedom.

[0078] refer to Figure 6 In Operation 650, the PCG solver can perform block-by-block computations of matrices and vectors. For example, the PCG solver's back-end processor can define a block of rows of a matrix. The back-end processor can perform block-by-block matrix and vector multiplication operations.

[0079] refer to Figure 6In operation 660, the PCG solver can sequentially output and accumulate the results of matrix and vector multiplication operations. For example, the back-end processor can output the result of the multiplication of the first block of a matrix and a vector as the first data. The back-end processor can shift the result data output after the first data to the location where the first data has already been stored. For example, the back-end processor can shift the result of the multiplication of each block other than the first block and a vector to the location where the output first data has already been stored. The back-end processor can simultaneously perform the output of the first data and the shifting of the result of the multiplication of each block other than the first block and a vector. In operation 660, the back-end processor can simultaneously perform the output of the first data and the shifting of the nth data, where n represents a natural number. The nth data can be the second data. For example, the back-end processor can shift the nth data to the location where the first data was stored before outputting it while simultaneously outputting the first data to a register where the accumulated result value is stored. The location where the first data was stored before outputting it can refer to a shift register. In operation 660, the first data and the nth data can be sequentially output and accumulated according to the calculation order of the back-end processor. Based on a preset time period, the back-end processor can control the output of the nth data point after the first data point. For example, the preset time period could be 0.1 seconds. When the preset time period is a low value, the back-end processor can execute the output of the first and nth data points in real time, which correspond to the calculated results of the components related to the six degrees of freedom of the electronic device embedded with the PCG solver.

[0080] Figure 7 It is a flowchart of mathematical operations performed by the PCG solver 200 according to various embodiments.

[0081] refer to Figure 7 The PCG solver can perform mathematical operations using matrices and vectors as input. In operation 701, the preprocessing unit of the PCG solver can receive the matrix S to which mathematical operations will be performed, along with vectors b and X0, as input from the memory array. For example, in operation 702, the PCG solver can obtain a product by multiplying matrix S by vector X0, and then subtract the product of matrix S and vector X0 from vector b. In other words, the PCG solver can obtain a value by multiplying matrix S by vector X0 and then by -1, and then adding that value to vector b. r0 can represent the residual vector.

[0082] refer to Figure 7 The PCG solver can perform mathematical operations using a preconditioner. In operation 703, the PCG solver can perform operations using the inverse of the preconditioner M. -1 Multiply by the residual vector r0. The inverse of the preconditioning factor M, M0. -1It can be used as a linear operator. For example, the inverse M of the preconditioning factor M. -1 The product of the residual vector r0 and the preconditioned residual vector z0 can be equal to the search direction vector p0. The search direction vector p0 can determine the direction of repetition in performing all repeated operations.

[0083] refer to Figure 7 In operation 704, the PCG solver can transform the transpose r of the k-th residual vector. k T The residual vector z multiplied by the k-th precondition k In operation 705, the PCG solver can multiply matrix S by the k-th search direction vector p. k .

[0084] refer to Figure 7 The PCG solver can compute a scalar α to determine the step size to take along the search direction toward the solution. For example, the PCG solver can compute the transpose p of the k-th search direction vector. k T Multiply by the product calculated in operation 705 (e.g., p) k T Sp k In operation 706, the PCG solver can divide the result value obtained in operation 704 by the value calculated by p. k T Sp k The value obtained. The quotient obtained in operation 706 can correspond to the k-th scalar α. k .

[0085] refer to Figure 7 The PCG solver can perform calculations related to the repeating vector X. For example, in operation 707, the PCG solver can compute the (k+1)th repeating vector X. k+1 The (k+1)th repeating vector X k+1 This can correspond to the k-th scalar α k and the k-th search direction vector p k The product is added to the k-th repeating vector X k The value obtained.

[0086] refer to Figure 7 The PCG solver can perform calculations related to the residual vector r. For example, in operation 708, the PCG solver can compute the (k+1)th residual vector r. k+1 The (k+1)th residual vector r k+1 This can correspond to the value obtained by: from the k-th residual vector r k Subtracting the k-th scalar α from the middle kMultiply by matrix S and then by the k-th search direction vector p k The product obtained.

[0087] refer to Figure 7 The PCG solver can solve for the (k+1)th residual vector r k+1 With threshold e threshold Comparisons are made. For example, in operation 709, the PCG solver can determine the (k+1)th residual vector r. k+1 Is the amplitude less than the threshold e? threshold .

[0088] refer to Figure 7 Given that the (k+1)th residual vector r has been determined k+1 The amplitude is greater than or equal to the threshold e threshold In this case, the PCG solver can perform operation 710. For example, the PCG solver can perform operation 710 by converting the inverse of the preconditioning factor M. -1 Multiplied by the (k+1)th residual vector r k+1 To calculate the residual vector z of the (k+1)th precondition. k+1 .

[0089] refer to Figure 7 The PCG solver can perform the calculation of the conjugate scalar β for the search direction. The PCG solver can also calculate the residual vector z of the (k+1)th precondition by transposing it. T k+1 Multiply by the (k+1)th residual vector r k+1 To calculate the (k+1)th result value. The PCG solver can do this by transposing the z-th residual vector of the k-th precondition. T k Multiply by the k-th residual vector r k To calculate the k-th result value. In operation 711, the PCG solver can calculate the k-th scalar β by dividing the (k+1)-th result value by the k-th result value. k .

[0090] refer to Figure 7 The PCG solver can perform the (k+1)th search direction vector p k+1 The calculation. For example, in operation 712, the PCG solver can calculate the (k+1)th search direction vector p as follows. k+1 : The residual vector z of the (k+1)th precondition k+1 Added by multiplying the k-th scalar βk by the k-th search direction vector p k The product obtained.

[0091] refer to Figure 7 In operation 713, the PCG solver can increment k by 1. In operation 709, the (k+1)th residual vector r is determined. k+1 The amplitude is less than the threshold ethreshold In this case, the PCG solver can perform operation 714 to output the (k+1)th repeating term vector X. k+1 .

[0092] Figure 8 These are example diagrams relating to mathematical operations performed by the PCG solver 200 and their output, according to various embodiments.

[0093] Figure 8 The examples shown may include mathematical operations performed by matrix-vector multiplier 220, and addition operations performed after matrix and vector multiplication. The mathematical operations 800 of the PCG solver may include arithmetic operations performed on matrices and matrices, matrices and vectors, vectors and vectors, and vectors and scalars.

[0094] refer to Figure 8 The back-end processor of the PCG solver (e.g., Figure 1 The backend processor 110 can configure matrix 810 according to factors. Matrix 810 can have a size of 75×75, and the larger shaded area defined by solid lines in row block 811 can depict a submatrix of size 15×15. The smaller shaded area in row block 811 can depict a submatrix of size 6×6 and can be associated with position and rotation among the components related to the six degrees of freedom. A dotted area defined by solid lines, other than the smaller shaded area in row block 811, can depict a submatrix defined by excluding the smaller shaded area from the 15×15 submatrix. The submatrix depicted by the dotted area can include three-axis components for IMU velocity, IMU acceleration, and IMU angular velocity. The description of the size and included areas of column block 812 can be replaced by the description of row block 811.

[0095] refer to Figure 8 PCG solvers (e.g., Figure 1 PCG solver 100 or Figure 2 The PCG solver 200 can perform calculations on each row block 811 of matrix 810 to quickly handle matrix and vector multiplication operations. The PCG solver can access the maximum size of data for each row on which it performs calculations, so that calculations can also be performed on transposed elements at once. Column blocks 812 can correspond to transposed elements, and thus can minimize mathematical operations. For mathematical operations performed on vectors, the PCG solver can access each element of the vector as many times as its size to perform mathematical operations sequentially, and then shift to access the next element.

[0096] Figure 8The example shown corresponds to the mathematical operations when the maximum track size is set to 5. Multiple multiplications can be performed on each element of the submatrix and vector, i.e., equal to the maximum track size. The calculation can also be performed on the transposed elements one at a time for each row. Reference Figure 8 The data corresponding to the results of multiplication and addition operations on matrix 810 and vector can be stored in output vector 830. In mathematical operations on row blocks, the product of elements of the submatrix and vector can be calculated, then addition can be performed on that product, and the resulting sum can be stored in the first output subvector 831. A single subvector can have a size of 15*1. The transpose elements can be calculated using vectors, and the resulting values ​​can then be stored in the second through fifth output subvectors 832 respectively. When performing the corresponding mathematical operation on each row, the calculation results for the transpose elements can be accumulated; and when the current row becomes the first row, the calculation can end, and the output vector for the corresponding row can be obtained as the result value.

[0097] Figure 9 These are example diagrams relating to mathematical simplification and result output of the PCG solver 200 according to various embodiments.

[0098] Figure 9 Example 800 shown can be used with Figure 8 The examples shown are the same and can correspond to examples used to accelerate mathematical operations. PCG solvers (e.g., Figure 1 PCG solver 100 or Figure 2 The PCG solver 200 can access and process fifteen line data points 911 of each row block component 910 (e.g., a 15×15 submatrix) at a time by accessing multiple lines. Processing at a time means performing mathematical operations at a time. Of the fifteen data points, the data related to the components related to the six degrees of freedom (excluding the IMU-related components) can be in 6×6 blocks, and therefore, the corresponding blocks can be collected and processed at once. (Reference) Figure 9 The PCG solver can process three lines at once, and can also perform mathematical operations on the transpose elements of the three lines simultaneously.

[0099] refer to Figure 9 As shown, the product calculated from matrices and vectors, along with their sums, can be stored in column vectors. Column vectors can have individual blocks of size 15x15. For example, the components 930 of a single block could correspond to position, rotation, linear velocity, acceleration, and angular velocity. R xn R yn and R znIt can be a component related to the six degrees of freedom. Among the components related to the six degrees of freedom, R xn It can be a rotational component relative to the x-axis, R yn It can be a rotational component relative to the y-axis and R zn It can be a rotational component relative to the z-axis. T xn T yn and T zn It can be a component related to the six degrees of freedom. Among the components related to the six degrees of freedom, T xn It can be the position component relative to the x-axis, T yn It can be the position component relative to the y-axis and T. zn It can be the position component relative to the z-axis.

[0100] refer to Figure 9 The component 930 of a single block can be a component related to the six degrees of freedom (specifically, a component related to the IMU). For example, V xn V yn and V zn It can be the linear velocity component among the six degrees of freedom components. Among the six degrees of freedom components, V... xn It can be a linear velocity component relative to the x-axis, V yn It can be a linear velocity component relative to the y-axis and V zn It can be a linear velocity component relative to the z-axis. Ba xn Ba yn and Ba zn It can be the acceleration-related component among the components related to the six degrees of freedom. Among the components related to the six degrees of freedom, Ba... xn It can be the acceleration component relative to the x-axis, Ba yn It can be the acceleration component relative to the y-axis and Ba zn This can be the acceleration component relative to the z-axis. (Bg) xn Bg yn and Bg zn It could be the component related to angular velocity among the components related to the six degrees of freedom. Among the components related to the six degrees of freedom, Bg... xn It can be the angular velocity component relative to the x-axis, Bg yn It can be the angular velocity component relative to the y-axis and Bg. zn It can be the angular velocity component relative to the z-axis.

[0101] A preconditioned conjugate gradient (PCG) solver embedded in an electronic device to perform simultaneous localization and mapping (SLAM) operations may include an image database, a factor graph database, and a back-end processor, wherein the back-end processor is configured to: receive images from the image database to perform relocalization; receive data from the factor graph database for computing components related to six degrees of freedom (DoF); construct a matrix including components related to the six degrees of freedom based on the received data; and load and rearrange the matrix and vectors to perform computations on each block of each row of the matrix and the vectors, then output first data and shift second data to the position of the first data.

[0102] The back-end processor of the PCG solver that performs SLAM operations can reuse as many elements as the block size in matrix and vector computations.

[0103] The back-end processor of the PCG solver that performs SLAM operations can compute only the diagonal elements of the matrix by using the properties of the scattering matrix (S matrix).

[0104] The back-end processor of the PCG solver that performs SLAM operations can also compute as many transpose elements as the maximum size of each row block in the computation performed on each block of each row of the constructed matrix.

[0105] The back-end processor of the PCG solver that performs SLAM operations can output the shifted second data and accumulate and output the second data together with the first data.

[0106] The back-end processor of the PCG solver that performs SLAM operations can simultaneously perform the accumulation of the first data and the shifting of the second data.

[0107] The back-end processor of the PCG solver that performs SLAM operations can perform the shifting of the second data by using shift registers.

[0108] The elements of the matrix constructed by the PCG solver performing the SLAM operation can include position, rotation, linear velocity, acceleration, and angular velocity components relative to the three axes of the electronic device.

[0109] The back-end processor of the PCG solver that performs SLAM operations can control the interval between the output of the first data and the output of the second data according to a preset time period.

[0110] The image database of the PCG solver that performs SLAM operations can store images received from the camera of an electronic device.

[0111] A method for accelerating simultaneous localization and mapping (SLAM) performed by a preconditioning conjugate gradient (PCG) solver embedded in an electronic device may include: receiving images from an image database to perform relocalization; receiving data from a factor graph database for computing components related to six degrees of freedom (DoF); constructing a matrix including components related to the six degrees of freedom based on the received data; and loading and rearranging the matrix and vectors to perform computations on each block of each row of the matrix and the vectors, then outputting first data and shifting second data to the position of the first data.

[0112] Accelerated Simultaneous Localization and Mapping (SLAM) methods performed by preconditioned conjugate gradient (PCG) solvers can further include reusing as many elements as the block size in computations performed on matrices and vectors.

[0113] The accelerated Simultaneous Localization and Mapping (SLAM) method performed by the preconditioned conjugate gradient (PCG) solver can further include: calculating only the diagonal elements of the matrix by using properties of the scattering matrix.

[0114] The accelerated Simultaneous Localization and Mapping (SLAM) method performed by a preconditional conjugate gradient (PCG) solver may further include: in the computation performed on each block of each row of the constructed matrix, computing as many transpose elements as the maximum size of each row block.

[0115] The accelerated Simultaneous Localization and Mapping (SLAM) method performed by the preconditioned conjugate gradient (PCG) solver may further include: outputting shifted second data, and accumulating and outputting the second data together with the first data.

[0116] The output of the second data in a Simultaneous Localization and Mapping (SLAM) method accelerated by a preconditioned conjugate gradient (PCG) solver can include simultaneously performing the accumulation of the first data and the shifting of the second data.

[0117] The method for accelerating simultaneous localization and mapping (SLAM) performed by a preconditioned conjugate gradient (PCG) solver may further include: performing a shift of second data by using a shift register.

[0118] The construction of matrices in accelerated Simultaneous Localization and Mapping (SLAM) methods performed by a preconditioned conjugate gradient (PCG) solver may include calculating the position, rotation, linear velocity, acceleration, and angular velocity components relative to the three axes of the electronic device.

[0119] The method for accelerating simultaneous localization and mapping (SLAM) performed by a preconditional conjugate gradient (PCG) solver may further include controlling the interval between the output of the first data and the output of the second data according to a preset time period.

[0120] The accelerated Simultaneous Localization and Mapping (SLAM) method, performed by a preconditioned conjugate gradient (PCG) solver, may further include storing images received from a camera of an electronic device in an image database.

[0121] As the processing speed of mathematical operations by backend processors increases, the overall time required to perform SLAM operations may decrease. In cases involving multiplication of matrices and vectors requiring extensive computation, backend processors can improve processing speed by utilizing properties of the S-matrix.

[0122] Back-end processors for preconditioned conjugate gradient (PCG) solvers can be configured in various devices to prevent data transmission and reception delays. With low-power, low-performance back-end processors, the size and power consumption of devices embedding these processors can be reduced.

[0123] While not limited thereto, the exemplary embodiments can be embodied as computer-readable code on a computer-readable recording medium. A computer-readable recording medium is any data storage device capable of storing data that can subsequently be read by a computer system. Examples of computer-readable recording media include read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices. The computer-readable recording medium can also be distributed across a network-coupled computer system, such that the computer-readable code is stored and executed in a distributed manner. Furthermore, the exemplary embodiments can be written as computer programs transmitted via a computer-readable transmission medium such as a carrier wave, and received and implemented in a general-purpose or special-purpose digital computer executing the program. Moreover, it should be understood that in the exemplary embodiments, one or more units of the above-described apparatus and devices may include circuits, processors, microprocessors, etc., and can execute computer programs stored on computer-readable media.

[0124] The foregoing exemplary embodiments are merely illustrative and should not be construed as limiting. This teaching can be readily applied to other types of devices. Furthermore, the description of the exemplary embodiments is intended to be illustrative and not to limit the scope of the claims, and many alternatives, modifications, and variations will be apparent to those skilled in the art.

Claims

1. An electronic device for performing simultaneous localization and mapping (SLAM) operations, the electronic device comprising: Image database; Factor graph database; and Processor, including: The preprocessing unit is configured as follows: Receive a map image from the image database in which the location of the electronic device is located; Receive inertial measurement unit (IMU) sensor data from the factor graph database; and The configuration matrix is ​​based on a fifteen-factor matrix, which is obtained by calculating the components of the electronic device related to six degrees of freedom (DoF). The matrix is ​​defined by multiple row blocks, which are divided into 15×15 submatrices, each of which has a 6×6 subsubmatrix and a remainder. The subsubmatrix is ​​related to the position and rotation of the components related to the six DoF, and the remainder includes the three-axis components for IMU sensor data. A multiplier unit is configured to perform a row-block multiplication with a vector for each of the plurality of row blocks of the matrix, and sequentially output the results of the row-block multiplications with the vector as first data; and The post-processing unit is configured to simultaneously shift the second data to the position where the first data has already been stored, the second data being the result of multiplying the vector by each subsequent row block in the plurality of row blocks.

2. The electronic device according to claim 1, wherein, The multiplier unit is also configured to reuse as many elements as the size of the row block in the computation of the matrix and the vector.

3. The electronic device according to claim 1, wherein, The preprocessing unit is also configured to compute only the diagonal elements of the scattering matrix by using properties of the scattering matrix derived from the map image and the IMU sensor data.

4. The electronic device according to claim 1, wherein, The multiplier unit is also configured to compute, in the computation performed on each row block of the matrix, the same number of transpose elements as the maximum size of each row block.

5. The electronic device according to claim 1, wherein, The post-processing unit is also configured to output the shifted second data and accumulate the second data and the first data.

6. The electronic device according to claim 5, wherein, The post-processing unit is also configured to simultaneously perform the accumulation of the first data and the second data and the shifting of the second data.

7. The electronic device according to claim 6, wherein, The post-processing unit is also configured to perform the shifting of the second data by using a shift register.

8. The electronic device according to claim 1, wherein, The elements of the matrix include position components, rotational components, linear velocity components, acceleration components, and angular velocity components relative to the three axes of the electronic device.

9. The electronic device according to claim 1, wherein, The processor is also configured to control the interval between the first output time of the first data and the second output time of the second data according to a preset time period.

10. The electronic device according to claim 1, wherein, The image database stores images received from the camera of the electronic device.

11. A method for accelerating simultaneous localization and mapping (SLAM) performed via an electronic device, the method comprising: Receive a map image from an image database, in which the location of the electronic device is determined; Receive inertial measurement unit (IMU) sensor data from the factor graph database of the electronic device; The configuration matrix is ​​based on a fifteen-factor matrix, which is obtained by calculating the components of the electronic device related to six degrees of freedom (DoF). The matrix is ​​defined by multiple row blocks, which are divided into 15×15 submatrices, each of which has a 6×6 subsubmatrix and a remainder. The subsubmatrix is ​​related to the position and rotation of the components related to the six DoF, and the remainder includes the three-axis components for IMU sensor data. For each of the plurality of row blocks in the matrix, a multiplication of the row block with the vector is performed, and the results of the multiplication of the row blocks with the vector in the plurality of row blocks are output sequentially as first data. At the same time, second data is shifted to the position where the first data has been stored. The second data is the result of the multiplication of each subsequent row block in the plurality of row blocks with the vector.

12. The method of claim 11, further comprising: In the computations performed on the matrix and the vector, the same number of elements as the row block size are reused.

13. The method of claim 11, further comprising: The scattering matrix is ​​derived from the map image and the IMU sensor data by calculating only the diagonal elements of the matrix using the properties of the scattering matrix.

14. The method of claim 11, further comprising: In the computation performed on each row block of the matrix, the same number of transpose elements as the maximum size of each row block are computed.

15. The method of claim 11, further comprising: Output the shifted second data, and accumulate the second data and the first data.

16. The method according to claim 15, wherein, The output of the second data includes simultaneously performing the accumulation of the first data and the second data and the shifting of the second data.

17. The method of claim 16, further comprising: The shifting of the second data is performed using a shift register.

18. The method according to claim 11, wherein, The construction of the matrix includes calculating the position component, rotation component, linear velocity component, acceleration component, and angular velocity component of the matrix relative to the three axes of the electronic device.

19. The method of claim 11, further comprising: The interval between the first output time of the first data and the second output time of the second data is controlled according to a preset time period.

20. The method of claim 11, further comprising: Images received from the camera of the electronic device are stored in the image database.

Images