Target object positioning method, system, device, electronic device and storage medium

Abstract

An embodiment of the present invention provides a method, system, device, electronic device and storage medium for positioning a target object. The method includes: obtaining environmental data and motion data collected by the target object; obtaining an original Hessian matrix and dimensional information of the original Hessian matrix, and performing marginalization processing on the original Hessian matrix according to the dimensional information to obtain a target Hessian matrix; solving the target Hessian matrix to obtain a plurality of sub-matrices; performing accelerated calculation according to the plurality of sub-matrices to obtain a Jacobian matrix and a residual matrix, wherein the Jacobian matrix is ​​a matrix used to characterize state information of a visual inertial fusion system, and the residual matrix is ​​a matrix used to optimize variable parameters; calculating the environmental data and the motion data according to the Jacobian matrix and the residual matrix to obtain positioning information of the target object.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a method for positioning a target object, a visual-inertial fusion system, a device for positioning a target object, an electronic device, and a computer-readable storage medium. Background Art

[0002] Visual Inertial System (VINS) is widely used in unmanned autonomous systems such as drones and self-driving cars. Although most modern unmanned autonomous systems use GNNS (Global Navigation Satellite System) and 4G / 5G technology as the basic positioning technology, the visual inertial system is still a very good auxiliary positioning technology for places with poor signal coverage such as tunnels and remote mountainous areas. Among them, although visual inertial system products have been applied in some applications, there are still some technical challenges for high-precision visual inertial systems. How to fuse the data of vision and inertial measurement unit (IMU) to develop a high-precision and high-robust visual inertial system is of great research and development significance.

[0003] For the visual inertial fusion system, its data processing can include pre-processing and post-processing. Pre-processing can be a method based on feature detection and tracking, and post-processing can be an optimization method based on nonlinear solution. Among them, post-processing enhances the robustness of the system and improves the positioning accuracy. Correspondingly, the amount of calculation of the system is also larger. Pre-processing can increase the data calculation speed, but the positioning accuracy is low. If the global optimization method is adopted, the system accuracy is high but the calculation efficiency is significantly reduced. Summary of the invention

[0004] The embodiments of the present invention provide a method, system, device, electronic device and computer-readable storage medium for positioning a target object, so as to solve or partially solve the problems of low computing efficiency and low positioning accuracy in the positioning process of a visual inertial fusion system.

[0005] The embodiment of the present invention discloses a method for locating a target object, comprising:

[0006] Obtain environmental data and motion data collected by the target object;

[0007] Obtaining an original Hessian matrix and dimension information of the original Hessian matrix, and performing marginalization processing on the original Hessian matrix according to the dimension information to obtain a target Hessian matrix;

[0008] Solving the target Hessian matrix to obtain several sub-matrices;

[0009] Performing accelerated calculation according to the plurality of sub-matrices to obtain a Jacobian matrix and a residual matrix, wherein the Jacobian matrix is ​​a matrix for representing state information of the visual inertial fusion system, and the residual matrix is ​​a matrix for optimizing variable parameters;

[0010] The environment data and the motion data are calculated according to the Jacobian matrix and the residual matrix to obtain the positioning information of the target object.

[0011] Optionally, the submatrix includes at least a first submatrix, a second submatrix, and a third submatrix, and performing accelerated calculation according to the plurality of submatrices to obtain a Jacobian matrix and a residual matrix includes:

[0012] Adding the first submatrix and the transposed first submatrix to obtain a first target matrix;

[0013] Perform a matrix operation according to the first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix;

[0014] Multiplying the first inverse matrix with the second submatrix and the third submatrix respectively to obtain a second target matrix corresponding to the second submatrix and a third target matrix corresponding to the third submatrix;

[0015] Performing a matrix operation according to the second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix;

[0016] Performing matrix operations using the third eigenvalue information of the diagonal matrix to obtain a Jacobian matrix;

[0017] A matrix operation is performed using the second inverse matrix and the third submatrix to obtain a residual matrix.

[0018] Optionally, performing a matrix operation according to the first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix includes:

[0019] Obtaining a first eigenvalue and a first eigenvector of the first target matrix;

[0020] A first matrix constructed by the first eigenvalues ​​and a second matrix constructed by the first eigenvectors are multiplied to obtain a first inverse matrix corresponding to the first target matrix.

[0021] Optionally, performing a matrix operation according to the second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix includes:

[0022] Obtaining a second eigenvalue and a second eigenvector of the second target matrix;

[0023] Multiplying a third matrix constructed by the second eigenvalues ​​and a fourth matrix constructed by the second eigenvectors to obtain a target vector and a target inverse vector corresponding to the second target matrix;

[0024] A square root operation is performed on each scalar in the target vector to obtain a diagonal matrix corresponding to the target vector, and a square root operation is performed on each scalar in the target inverse vector to obtain a second inverse matrix corresponding to the diagonal matrix.

[0025] Optionally, the performing matrix operation using the third eigenvalue information of the diagonal matrix to obtain a Jacobian matrix includes:

[0026] Obtaining a third eigenvector of the diagonal matrix;

[0027] The diagonal matrix and the third eigenvector are multiplied to obtain a Jacobian matrix.

[0028] Optionally, the using the second inverse matrix and the third submatrix to perform a matrix operation to obtain a residual matrix includes:

[0029] Obtaining a fourth eigenvector of the second inverse matrix;

[0030] The second inverse matrix, the fourth eigenvector and the third submatrix are multiplied to obtain a residual matrix.

[0031] Optionally, the dimensional information includes first dimensional information to be marginalized and second dimensional information to be retained, and performing marginalization processing on the original Hessian matrix according to the dimensional information to obtain a target Hessian matrix includes:

[0032] The first dimensional information and the second dimensional information are used to perform marginalization processing on the original Hessian matrix to construct a target Hessian matrix.

[0033] The embodiment of the present invention further discloses a visual-inertial fusion system, which includes a visual and inertial measurement unit, a control instruction memory, a tensor data memory, a tensor arithmetic logic operation array, and a tensor acceleration calculation control unit; the tensor data memory stores environmental data and motion data collected by the visual and inertial measurement unit; wherein,

[0034] The tensor acceleration calculation control unit is used to transmit the positioning operation instruction to the tensor arithmetic logic operation array in response to receiving the positioning operation instruction transmitted by the control instruction memory;

[0035] The tensor arithmetic logic operation array is used to obtain an original Hessian matrix and dimension information for the original Hessian matrix from the tensor data memory according to the positioning operation instruction, and perform marginalization processing on the original Hessian matrix according to the dimension information to obtain a target Hessian matrix; solve the target Hessian matrix to obtain a plurality of sub-matrices; perform accelerated calculation according to the plurality of sub-matrices to obtain a Jacobian matrix and a residual matrix; calculate the environmental data and the motion data according to the Jacobian matrix and the residual matrix to obtain the positioning information of the target object, wherein the Jacobian matrix is ​​a matrix used to characterize the state information of the visual inertial fusion system, and the residual matrix is ​​a matrix used to optimize variable parameters;

[0036] The tensor data storage device is used to store the target Hessian matrix, the Jacobian matrix, the residual matrix and the positioning information.

[0037] Optionally, the tensor arithmetic logic operation array includes a calculation instruction parsing unit, a plurality of arithmetic logic units and an adder tree;

[0038] The computing instruction parsing unit is used to parse the positioning operation instruction, obtain the environmental data, the motion data and the original Hessian matrix from the tensor data memory, and obtain dimension information for the original Hessian matrix;

[0039] The arithmetic logic unit is used to add the first submatrix and the transposed first submatrix to obtain a first target matrix; perform matrix operations according to the first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix; multiply the first inverse matrix with the second submatrix and the third submatrix respectively to obtain a second target matrix corresponding to the second submatrix and a third target matrix corresponding to the third submatrix; perform matrix operations according to the second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix; perform matrix operations using the third characteristic information of the diagonal matrix to obtain a Jacobian matrix; perform matrix operations using the second inverse matrix and the third submatrix to obtain a residual matrix;

[0040] The adder tree is used to accelerate operations when performing matrix multiplication.

[0041] Optionally, the arithmetic logic unit includes a first arithmetic logic unit and several second arithmetic logic units; wherein the first arithmetic logic unit is used to perform at least one operation of addition, subtraction, multiplication, division and square root, and the second arithmetic logic unit is used to perform at least one operation of addition, subtraction and multiplication.

[0042] The embodiment of the present invention further discloses a device for locating a target object, comprising:

[0043] A data acquisition module is used to acquire environmental data and motion data collected by the target object;

[0044] A Hessian matrix acquisition module, used to obtain an original Hessian matrix and dimension information of the original Hessian matrix, and perform marginalization processing on the original Hessian matrix according to the dimension information to obtain a target Hessian matrix;

[0045] A solution module, used for solving the target Hessian matrix to obtain a plurality of sub-matrices;

[0046] An accelerated calculation module, used for performing accelerated calculation according to the plurality of sub-matrices to obtain a Jacobian matrix and a residual matrix, wherein the Jacobian matrix is ​​a matrix for representing state information of the visual inertial fusion system, and the residual matrix is ​​a matrix for optimizing variable parameters;

[0047] A positioning module is used to calculate the environmental data and the motion data according to the Jacobian matrix and the residual matrix to obtain the positioning information of the target object.

[0048] Optionally, the sub-matrix includes at least a first sub-matrix, a second sub-matrix and a third sub-matrix, and the accelerated computing module is specifically used for:

[0049] Adding the first submatrix and the transposed first submatrix to obtain a first target matrix;

[0050] Perform a matrix operation according to the first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix;

[0051] Multiplying the first inverse matrix with the second submatrix and the third submatrix respectively to obtain a second target matrix corresponding to the second submatrix and a third target matrix corresponding to the third submatrix;

[0052] Performing a matrix operation according to the second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix;

[0053] Performing matrix operations using the third eigenvalue information of the diagonal matrix to obtain a Jacobian matrix;

[0054] A matrix operation is performed using the second inverse matrix and the third submatrix to obtain a residual matrix.

[0055] Optionally, the accelerated computing module is specifically used for:

[0056] Obtaining a first eigenvalue and a first eigenvector of the first target matrix;

[0057] A first matrix constructed by the first eigenvalues ​​and a second matrix constructed by the first eigenvectors are multiplied to obtain a first inverse matrix corresponding to the first target matrix.

[0058] Optionally, the accelerated computing module is specifically used for:

[0059] Obtaining a second eigenvalue and a second eigenvector of the second target matrix;

[0060] Multiplying a third matrix constructed by the second eigenvalues ​​and a fourth matrix constructed by the second eigenvectors to obtain a target vector and a target inverse vector corresponding to the second target matrix;

[0061] A square root operation is performed on each scalar in the target vector to obtain a diagonal matrix corresponding to the target vector, and a square root operation is performed on each scalar in the target inverse vector to obtain a second inverse matrix corresponding to the diagonal matrix.

[0062] Optionally, the accelerated computing module is specifically used for:

[0063] Obtaining a third eigenvector of the diagonal matrix;

[0064] The diagonal matrix and the third eigenvector are multiplied to obtain a Jacobian matrix.

[0065] Optionally, the accelerated computing module is specifically used for:

[0066] Obtaining a fourth eigenvector of the second inverse matrix;

[0067] The second inverse matrix, the fourth eigenvector and the third submatrix are multiplied to obtain a residual matrix.

[0068] Optionally, the dimensional information includes first dimensional information to be marginalized and second dimensional information to be retained, and the Hessian matrix acquisition module is specifically used for:

[0069] The first dimensional information and the second dimensional information are used to perform marginalization processing on the original Hessian matrix to construct a target Hessian matrix.

[0070] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0071] The memory is used to store computer programs;

[0072] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0073] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0074] The embodiments of the present invention include the following advantages:

[0075] In an embodiment of the present invention, when the target object is positioned, the collected environmental data and motion data can be obtained, and then the Hessian matrix used for positioning operation and the dimensional information of the original Hessian matrix can be obtained, and the original Hessian matrix can be marginalized according to the dimensional information to obtain the target Hessian matrix, and then the target Hessian matrix can be solved to obtain several sub-matrices, and then accelerated calculations are performed based on the several sub-matrices to obtain the Jacobian matrix and the residual matrix, the Jacobian matrix is ​​a matrix used to characterize the state information of the visual inertial fusion system, and the residual matrix is ​​a matrix used to optimize variable parameters, and then the environmental data and motion data are calculated according to the Jacobian matrix and the residual matrix to obtain the positioning information of the target object, so that in the process of positioning, especially when auxiliary positioning is performed by the visual inertial fusion system, the corresponding Jacobian matrix and the residual matrix are obtained by marginalizing and accelerating the calculation of the Hessian matrix in the system, which can effectively control the amount of calculation of the system and improve the calculation efficiency while ensuring the positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 is a flowchart of a method for locating a target object provided in an embodiment of the present invention;

[0077] Figure 2 is a schematic diagram of the structure of a visual-inertial fusion system provided in an embodiment of the present invention;

[0078] Figure 3 is a schematic diagram of the structure of a tensor acceleration calculation control unit provided in an embodiment of the present invention;

[0079] Figure 4 is a schematic diagram of the structure of a tensor arithmetic logic operation array provided in an embodiment of the present invention;

[0080] Figure 5 is a schematic diagram of the structure of a tensor data storage device provided in an embodiment of the present invention;

[0081] Figure 6 is a schematic diagram of a Hessian matrix provided in an embodiment of the present invention;

[0082] Figure 7 is a structural block diagram of a visual-inertial fusion system provided in an embodiment of the present invention;

[0083] Figure 8 is a structural block diagram of a target object positioning device provided in an embodiment of the present invention;

[0084] Fig. 9 It is a block diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0085] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0086] As an example, for some positioning scenarios, such as driverless cars driving in underground parking lots and flying cars moving in remote mountainous areas, when there is no positioning signal or the 4G / 5G signal is missing or of poor quality, it is necessary to rely on the visual inertial fusion system for positioning. Among them, the current implementation schemes of the visual inertial fusion system are mainly divided into two methods: microprocessors and domain-specific processors (ASIP, Application Specific Instructionset Processor). The general microprocessor implementation scheme has low cost and short development cycle. However, the current visual inertial fusion system has a low data throughput rate for 1080P resolution and cannot meet the performance requirements of the visual inertial fusion system for high-level autonomous driving application scenarios.

[0087] For the visual inertial fusion system, its data processing can include pre-processing and post-processing. The pre-processing can be a method based on feature detection and tracking, and the post-processing can be an optimization method based on nonlinear solution. Among them, the post-processing enhances the robustness of the system and improves the positioning accuracy. Correspondingly, the amount of calculation of the system is also larger. The pre-processing can improve the data calculation speed, but the positioning accuracy is low. If the global optimization method is adopted, the accuracy of the system is high but the calculation efficiency is significantly reduced. Therefore, the sliding window method is introduced to optimize a specific number of frames each time, which ensures both accuracy and efficiency. During the sliding process of the window, the new image frame and IMU data update and replace the old data. Such marginalization operation can better utilize the information of historical image frames and IMU. In the above process, the marginalization calculation of the Hessian matrix accounts for the largest proportion of the post-processing calculation in the VINS system. Therefore, improving the calculation efficiency of the marginalization of the Hessian matrix is ​​the key to improving the system throughput.

[0088] In this regard, one of the core invention points of the present invention is to construct an edge computing acceleration structure for a visual-inertial fusion system, and accelerate the edge computing of the Hessian matrix through the edge computing acceleration structure, thereby improving the computing efficiency of the largest processing calculation in the positioning operation, thereby improving the throughput of the visual-inertial fusion system by improving the computing efficiency of the Hessian matrix edge computing.

[0089] Reference Figure 1 , shows a flowchart of a method for locating a target object provided in an embodiment of the present invention, which may specifically include the following steps:

[0090] Step 101, obtaining environmental data and motion data collected by the target object;

[0091] Optionally, the target object may be a vehicle, flying car, drone, ship or other object that needs to be positioned, and corresponding sensors may be deployed on the target object to achieve positioning. In the case of poor positioning signals or network signals, the target object cannot effectively position itself, thereby affecting the movement or parking of the target object. In this regard, a visual-inertial fusion system may be deployed on the target object, and auxiliary positioning may be performed based on the visual-inertial fusion system when the positioning signal and / or network signal of the target object is abnormal.

[0092] In a specific implementation, the visual inertial fusion system can respond to signal abnormality instructions (including positioning signal abnormality, network signal abnormality), and then the environmental data and motion data collected by the visual and inertial measurement units. Optionally, the visual inertial fusion system can be a fusion system of both a binocular camera and a visual inertial system, and the system can include visual sensors and inertial measurement sensors. For visual sensors, it can be various cameras, which record sensor data in the form of images. Generally speaking, the frequency is low (24Hz to 60Hz). We can estimate its motion through multi-view geometry. In fusion, its advantage is that the drift is small when it is close to no motion, and the posture can be directly estimated; for inertial measurement sensors, it can measure its own acceleration and angular velocity through accelerometers and gyroscopes. Generally speaking, the frequency is very high (100Hz-1kHz), and the displacement and rotation of its own motion are obtained by integrating the acceleration and angular velocity, so that when the positioning signal and / or network signal of the target object is abnormal, the visual inertial fusion system can process the collected environmental data and motion data to achieve auxiliary positioning.

[0093] Step 102, obtaining an original Hessian matrix and dimension information of the original Hessian matrix, and performing marginalization processing on the original Hessian matrix according to the dimension information to obtain a target Hessian matrix;

[0094] In an embodiment of the present invention, before processing the environmental data and motion data, the visual inertial fusion system may first perform edge calculation acceleration on the Hessian matrix, and then process the environmental data and motion data according to the acceleration result to achieve the positioning of the target object.

[0095] Among them, for the visual-inertial fusion system, it can be a system based on a multi-core processor chip, and the multi-core processor chip system includes a processor system (PS, Processing System) and a programmable logic unit (PL, Programmable Logic). The visual-inertial fusion system may include a tensor calculation acceleration unit, which is used to complete the control of storage and tensor calculation, and can be specifically located in the PL. The visual-inertial fusion system may specifically include a control instruction memory, a tensor data memory, a tensor arithmetic logic operation array, and a tensor calculation acceleration control unit. Optionally, the tensor calculation unit is divided into two organizational forms, one is 64-way half-precision (FP16, Floating Point 16bits), and the other is 32-way single precision (FP32, Floating Point 32bits), and the precision of FP32 can be higher than that of FP16.

[0096] In one example, referring to Figure 2, showing a schematic diagram of the structure of the visual-inertial fusion system provided in an embodiment of the present invention. The components involved in tensor calculation acceleration in the visual-inertial fusion system may include ① a control instruction memory, ② a tensor data memory, ③ a tensor arithmetic logic operation array, ④ a tensor calculation acceleration control unit and ⑤ a data transmission unit, etc. Among them, ① the control instruction memory is used to store operation control instruction data, which is generated offline by the assembly interpreter and issued by the PS; ② the tensor data memory can be used to store the original Hessian matrix, the Hessian matrix that needs to be marginalized, the Jacobian matrix, and the residual term, which can include several URAM components, and the URAM component can be a PL internal memory component; ③ the tensor arithmetic logic operation array can be used to complete various tensor calculation operations, including scalar division, square root, etc., 1D tensor addition, subtraction, multiplication, etc., and the operation of 2D tensors can be completed through the loop operation of 1D tensors, which can include multiple organizational forms, such as FP16 and FP32; ④ the tensor calculation acceleration control unit can be used to complete the control of storage and tensor calculation; ⑤ the data transmission unit can be used to complete the data interaction between the calculator on the PL side of the programmable logic unit and the DRAM (Dynamic Random Access Memory) on the PS side of the processor system, which supports the Load operation of PS side data to PL, and also supports the Fetch operation of PS obtaining calculation results from PL. The amount of computation can be effectively controlled by using marginalization calculation method. Selectively deleting the next newest frame or the oldest frame in the sliding window can ensure that the frames in the sliding window have sufficient parallax. When deleting the oldest frame, the constraints of the old frame data on other frames (i.e., prior information) should be retained in the form of marginalization constraints.

[0097] Reference Figure 3, shows a schematic diagram of the structure of the tensor acceleration computing control unit provided in an embodiment of the present invention. For the tensor acceleration computing control unit, it may include ① a control instruction unit (PC, Program Counter), ② a data transmission control interface unit, and ③ a tensor computing control unit. Among them, ① the output of the control instruction unit is the input of the control instruction memory address port, and the control instruction unit can point to different addresses of the instruction memory according to different operation types. As mentioned above, the content of the control instruction memory is generated by an offline assembly interpreter, which can cover all marginalized computing operations; the instruction output by the control instruction memory will directly drive two units, namely ② a data transmission control interface unit and ③ a tensor computing control unit; ② the data transmission control unit can be used to control the data transmission module to complete the interface with the data transmission module; ③ the tensor computing control unit can be used to obtain the corresponding data, the original Hessian matrix, etc. from the tensor data memory, and perform Hessian matrix calculation and positioning operations through the tensor arithmetic logic operation array. In addition, the tensor acceleration computing control unit may also include a selector, which may be used to determine whether the output result of the control instruction parsing unit is transmitted to ② the data transmission control interface unit or ③ the tensor computing control unit according to the instruction type.

[0098] Reference Figure 4 , shows a schematic diagram of the structure of the tensor arithmetic logic operation array provided in an embodiment of the present invention. For the tensor arithmetic logic operation array, it may include components such as ① a calculation instruction parsing unit, several ②-③ arithmetic logic units and ④ an adder tree, and the arithmetic logic unit may include a homogeneous arithmetic logic unit and a heterogeneous arithmetic logic unit. Among them, ① the calculation instruction parsing unit can be used to obtain the calculation instruction from the tensor acceleration calculation control unit, and select the corresponding data in the tensor data memory to be transferred to the ② arithmetic logic unit or the ③ arithmetic logic unit for calculation according to the calculation instruction, and after the calculation is completed, the corresponding calculation result can be selected according to the calculation instruction and written into the tensor data memory. Among them, for the tensor arithmetic logic operation array in the form of 64-way half-precision or 32-way single-precision organization, a total of 64 or 32 arithmetic logic units are required. Considering the actual algorithm and PL resources, the arithmetic logic units can be divided into 2 types, namely ② arithmetic logic unit 0 and ③ arithmetic logic unit N. Among them, only one ② arithmetic logic unit 0 is deployed, which can support addition, subtraction, multiplication, division, and square root operations. The remaining arithmetic logic units are all ③ arithmetic logic units N, which only support addition, subtraction, and multiplication operations. At the same time, ④ addition trees are deployed in the tensor arithmetic logic operation array to accelerate matrix multiplication. In addition, the tensor arithmetic logic operation array also includes a selector, which is used to determine which operation result is transmitted to which arithmetic logic unit according to the instruction type, and to determine which operation result of the arithmetic logic unit is transmitted to the tensor data memory according to the instruction type.

[0099] Reference Figure 5 , shows a schematic diagram of the structure of the tensor data memory provided in an embodiment of the present invention. For the tensor data memory, it can be composed of 16 ② basic storage units, and each ② basic storage unit is composed of 2 ① URAM (UltraRAM). ① URAM can be a PL internal memory component, which is 64bitx4096 in size, has a read port and a write port, and each address stores 4 half-precision floating point numbers (FP16) or 2 single-precision floating point numbers (FP32). It can be understood that, considering the actual computing requirements, two ① URAMs are used to form the ② basic storage unit, and each ② basic storage unit has a write port and two read ports. The write ports of the two ① URAMs are connected together to ensure that the data stored in the two ① URAMs are completely consistent; the read ports of the two ① URAMs are directly used as the two read ports of the ② basic storage unit to ensure that 2 different numbers can be read from different addresses of the ② basic storage unit at the same time for calculation. Optionally, the tensor data memory is composed of 16 ② basic storage units, and the effective storage data capacity is 1024bit x 4096. The tensor data memory is controlled by the tensor acceleration computing control unit to have its read and write addresses and read and write enables, to provide operands for the tensor arithmetic logic array, and to store the calculation results of the tensor arithmetic logic array.

[0100] Through the accelerated computing structure of the visual-inertial fusion system, the marginalized computing acceleration of the Hessian matrix of the visual-inertial fusion system can be effectively performed, thereby effectively controlling the amount of computing and improving computing efficiency while ensuring the positioning accuracy of the system.

[0101] For the marginalization acceleration calculation of the Hessian matrix, three stages can be included: first, constructing a marginalizable Hessian matrix, second, solving the Schur complement of the Hessian matrix, and third, updating the Jacobian matrix and the residual matrix. Among them, for constructing a marginalizable Hessian matrix, the first dimension information to be marginalized and the second dimension information to be retained can be obtained, and then the first dimension information and the second dimension information can be used to marginalize the original Hessian matrix to construct the target Hessian matrix.

[0102] In one example, referring to Figure 6 , a schematic diagram of the Hessian matrix provided in an embodiment of the present invention is shown. The original Hessian matrix can be loaded into the tensor data storage device, and the approximate second-order state information (i.e., the Hessian matrix after marginalization) can be quickly obtained through marginalization calculation. Specifically, the PS can load matrix data to the PL according to the dimension m that needs to be marginalized and the dimension n that needs to be retained to obtain the target Hessian matrix.

[0103] Step 103, solving the target Hessian matrix to obtain a plurality of sub-matrices;

[0104] like Figure 6 As shown, the target Hessian matrix can be solved to obtain different sub-matrices such as Amm matrix, Amr matrix, Arm matrix and Arr matrix, so as to realize accelerated calculation of Hessian matrix marginalization according to different sub-matrices.

[0105] Step 104, performing accelerated calculation according to the plurality of sub-matrices to obtain a Jacobian matrix and a residual matrix, wherein the Jacobian matrix is ​​a matrix for representing state information of the visual inertial fusion system, and the residual matrix is ​​a matrix for optimizing variable parameters;

[0106] In a specific implementation, the Jacobian matrix and the residual matrix can be obtained by performing accelerated calculation of marginalization on the Hessian matrix. Specifically, the submatrix includes at least a first submatrix, a second submatrix and a third submatrix, then the first submatrix and the transposed first submatrix can be added to obtain a first target matrix, and a matrix operation can be performed according to the first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix, and then the first inverse matrix is ​​multiplied with the second submatrix and the third submatrix respectively to obtain a second target matrix corresponding to the second submatrix and a third target matrix corresponding to the third submatrix, and then a matrix operation can be performed according to the second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix, and then the third characteristic information of the diagonal matrix can be used to perform matrix operations to obtain a Jacobian matrix, and the second inverse matrix and the third submatrix can be used to perform matrix operations to obtain a residual matrix, so that the corresponding Jacobian matrix and the residual matrix can be obtained by performing accelerated calculation of marginalization on the Hessian matrix, which can effectively control the amount of calculation of the system and improve the calculation efficiency while ensuring the positioning accuracy.

[0107] In an optional embodiment, the marginalization acceleration calculation of the Hessian matrix may specifically include:

[0108] The first submatrix and the transposed first submatrix are added to obtain the first target matrix, and then the first eigenvalue and the first eigenvector of the first target matrix are obtained, and then the first matrix constructed by the first eigenvalue and the second matrix constructed by the first eigenvector are multiplied to obtain the first inverse matrix corresponding to the first target matrix. After obtaining the first inverse matrix, the first inverse matrix can be multiplied with the second submatrix and the third submatrix respectively to obtain the second target matrix corresponding to the second submatrix and the third target matrix corresponding to the third submatrix, and then the second eigenvalue and the second eigenvector of the second target matrix can be obtained, and the third matrix constructed by the second eigenvalue and the fourth matrix constructed by the second eigenvector are multiplied to obtain the target vector and the target inverse vector corresponding to the second target matrix, and then each scalar in the target vector can be square rooted to obtain the diagonal matrix corresponding to the target vector, and each scalar in the target inverse vector can be square rooted to obtain the second inverse matrix corresponding to the diagonal matrix. After obtaining the diagonal matrix and the second inverse matrix corresponding to the diagonal matrix, the third eigenvector of the diagonal matrix can be obtained, and the diagonal matrix and the third eigenvector can be multiplied to obtain the Jacobian matrix. At the same time, the fourth eigenvector of the second inverse matrix can be obtained, and the second inverse matrix, the fourth eigenvector and the third submatrix can be multiplied to obtain the residual matrix.

[0109] In one example, the first submatrix, the second submatrix, the third submatrix, and the fourth submatrix, etc., can be matrix elements of the target Hessian matrix, which can correspond to the Amm matrix, the Amr matrix, the Arm matrix, and the Arr matrix in the target Hessian matrix, respectively. In the accelerated calculation process of marginalizing the Hessian matrix, the Amm matrix and the transposed Amm matrix in the target Hessian matrix can be added first to obtain the target Amm matrix, and then the first eigenvalue and the first eigenvector of the target Amm matrix can be calculated in PS, and the matrices respectively formed by the first eigenvalue and the second eigenvector are multiplied to obtain the Amm_inv matrix. After obtaining the inverse matrix Amm_inv matrix of the target Amm matrix, it can be multiplied with the Amr matrix and the Arm matrix in the target Hessian matrix, respectively, to obtain the updated Hessian matrix A and Hessian matrix B with dimensions of n*n. Next, the eigenvalues ​​and eigenvectors of the Hessian matrix A can be calculated respectively, and then the matrices composed of the eigenvalues ​​and eigenvectors can be multiplied again to obtain the S vector and S_inv inverse vector corresponding to the Hessian matrix A. Then, each scalar in the S vector and the S_inv inverse vector can be square rooted point by point to construct a diagonal matrix S_sqrt composed of an eigenvalue array and an inverse matrix S_sqrt_inv corresponding to the diagonal matrix S_sqrt. Then, the eigenvectors of the diagonal matrix S_sqrt can be obtained, and the diagonal matrix S_sqrt can be multiplied with its corresponding eigenvector to obtain a Jacobian matrix that can be used to characterize the state information of the current fusion system. At the same time, the inverse matrix S_sqrt_inv can be multiplied with its corresponding eigenvector and the Hessian matrix B to obtain a residual matrix, so that the corresponding Jacobian matrix and residual matrix can be obtained by performing marginalized acceleration calculation through the Hessian matrix, which can effectively control the amount of calculation of the system and improve the calculation efficiency while ensuring the positioning accuracy.

[0110] After completing the Hessian matrix marginalization calculation acceleration through the above process, the collected environmental data and motion data can be processed based on the Jacobian matrix and residual matrix obtained after the calculation acceleration to realize the positioning of the target object. Therefore, when the positioning signal and / or network signal of the target object are abnormal, auxiliary positioning can be performed through the visual inertial fusion system.

[0111] Step 105 , calculating the environment data and the motion data according to the Jacobian matrix and the residual matrix to obtain the positioning information of the target object.

[0112] After completing the acceleration of the Hessian matrix marginalization calculation, the target object can be calculated based on the obtained Jacobian matrix and the residual matrix for the environmental data and motion data collected by the visual and inertial measurement units to obtain the positioning information of the target object. Optionally, the calculation of the positioning process belongs to the relevant prior art and will not be repeated here.

[0113] It should be noted that the embodiments of the present invention include but are not limited to the above examples. It is understandable that based on the teachings of the embodiments of the present invention, those skilled in the art may also adopt other methods to perform adaptive adjustments, and the present invention is not limited to this.

[0114] In an embodiment of the present invention, when the target object is positioned, the collected environmental data and motion data can be obtained, and then the Hessian matrix used for positioning operation and the dimensional information of the original Hessian matrix can be obtained, and the original Hessian matrix can be marginalized according to the dimensional information to obtain the target Hessian matrix, and then the target Hessian matrix can be solved to obtain several sub-matrices, and then accelerated calculations are performed based on the several sub-matrices to obtain the Jacobian matrix and the residual matrix, the Jacobian matrix is ​​a matrix used to characterize the state information of the visual inertial fusion system, and the residual matrix is ​​a matrix used to optimize variable parameters, and then the environmental data and motion data are calculated according to the Jacobian matrix and the residual matrix to obtain the positioning information of the target object, so that in the process of positioning, especially when auxiliary positioning is performed by the visual inertial fusion system, the corresponding Jacobian matrix and the residual matrix are obtained by marginalizing and accelerating the calculation of the Hessian matrix in the system, which can effectively control the amount of calculation of the system and improve the calculation efficiency while ensuring the positioning accuracy.

[0115] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0116] Reference Figure 7 , shows a structural block diagram of a visual inertial fusion system provided in an embodiment of the present invention, which may specifically include a visual and inertial measurement unit, a control instruction memory, a tensor data memory, a tensor arithmetic logic operation array, and a tensor acceleration calculation control unit; the tensor data memory stores the environmental data and motion data collected by the visual and inertial measurement unit; wherein,

[0117] The tensor acceleration calculation control unit is used to transmit the positioning operation instruction to the tensor arithmetic logic operation array in response to receiving the positioning operation instruction transmitted by the control instruction memory;

[0118] The tensor arithmetic logic operation array is used to obtain an original Hessian matrix and dimension information for the original Hessian matrix from the tensor data memory according to the positioning operation instruction, and perform marginalization processing on the original Hessian matrix according to the dimension information to obtain a target Hessian matrix; solve the target Hessian matrix to obtain a plurality of sub-matrices; perform accelerated calculation according to the plurality of sub-matrices to obtain a Jacobian matrix and a residual matrix; calculate the environmental data and the motion data according to the Jacobian matrix and the residual matrix to obtain the positioning information of the target object, wherein the Jacobian matrix is ​​a matrix used to characterize the state information of the visual inertial fusion system, and the residual matrix is ​​a matrix used to optimize variable parameters;

[0119] The tensor data storage device is used to store the target Hessian matrix, the Jacobian matrix, the residual matrix and the positioning information.

[0120] In an optional embodiment, the tensor arithmetic logic operation array includes a calculation instruction parsing unit, a plurality of arithmetic logic units and an adder tree;

[0121] The computing instruction parsing unit is used to parse the positioning operation instruction, obtain the environmental data, the motion data and the original Hessian matrix from the tensor data memory, and obtain dimension information for the original Hessian matrix;

[0122] The arithmetic logic unit is used to add the first submatrix and the transposed first submatrix to obtain a first target matrix; perform matrix operations according to the first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix; multiply the first inverse matrix with the second submatrix and the third submatrix respectively to obtain a second target matrix corresponding to the second submatrix and a third target matrix corresponding to the third submatrix; perform matrix operations according to the second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix; perform matrix operations using the third characteristic information of the diagonal matrix to obtain a Jacobian matrix; perform matrix operations using the second inverse matrix and the third submatrix to obtain a residual matrix;

[0123] The adder tree is used to accelerate operations when performing matrix multiplication.

[0124] In an optional embodiment, the arithmetic logic unit includes a first arithmetic logic unit and several second arithmetic logic units; wherein the first arithmetic logic unit is used to perform at least one operation of addition, subtraction, multiplication, division and square root, and the second arithmetic logic unit is used to perform at least one operation of addition, subtraction and multiplication.

[0125] Reference Figure 8 , shows a structural block diagram of a target object positioning device provided in an embodiment of the present invention, which may specifically include the following modules:

[0126] The data acquisition module 801 is used to acquire the environmental data and motion data collected by the target object;

[0127] A Hessian matrix acquisition module 802 is used to obtain an original Hessian matrix and dimension information of the original Hessian matrix, and perform marginalization processing on the original Hessian matrix according to the dimension information to obtain a target Hessian matrix;

[0128] A solution module 803 is used to solve the target Hessian matrix to obtain a plurality of sub-matrices;

[0129] An accelerated calculation module 804 is used to perform accelerated calculation according to the plurality of sub-matrices to obtain a Jacobian matrix and a residual matrix, wherein the Jacobian matrix is ​​a matrix used to characterize state information of the visual inertial fusion system, and the residual matrix is ​​a matrix used to optimize variable parameters;

[0130] The positioning module 805 is used to calculate the environmental data and the motion data according to the Jacobian matrix and the residual matrix to obtain the positioning information of the target object.

[0131] In an optional embodiment, the sub-matrix includes at least a first sub-matrix, a second sub-matrix and a third sub-matrix, and the accelerated calculation module 804 is specifically used for:

[0132] Adding the first submatrix and the transposed first submatrix to obtain a first target matrix;

[0133] Perform a matrix operation according to the first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix;

[0134] Multiplying the first inverse matrix with the second submatrix and the third submatrix respectively to obtain a second target matrix corresponding to the second submatrix and a third target matrix corresponding to the third submatrix;

[0135] Performing a matrix operation according to the second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix;

[0136] Performing matrix operations using the third eigenvalue information of the diagonal matrix to obtain a Jacobian matrix;

[0137] A matrix operation is performed using the second inverse matrix and the third submatrix to obtain a residual matrix.

[0138] In an optional embodiment, the accelerated computing module 804 is specifically used for:

[0139] Obtaining a first eigenvalue and a first eigenvector of the first target matrix;

[0140] A first matrix constructed by the first eigenvalues ​​and a second matrix constructed by the first eigenvectors are multiplied to obtain a first inverse matrix corresponding to the first target matrix.

[0141] In an optional embodiment, the accelerated computing module 804 is specifically used for:

[0142] Obtaining a second eigenvalue and a second eigenvector of the second target matrix;

[0143] Multiplying a third matrix constructed by the second eigenvalues ​​and a fourth matrix constructed by the second eigenvectors to obtain a target vector and a target inverse vector corresponding to the second target matrix;

[0144] A square root operation is performed on each scalar in the target vector to obtain a diagonal matrix corresponding to the target vector, and a square root operation is performed on each scalar in the target inverse vector to obtain a second inverse matrix corresponding to the diagonal matrix.

[0145] In an optional embodiment, the accelerated computing module 804 is specifically used for:

[0146] Obtaining a third eigenvector of the diagonal matrix;

[0147] The diagonal matrix and the third eigenvector are multiplied to obtain a Jacobian matrix.

[0148] In an optional embodiment, the accelerated computing module 804 is specifically used for:

[0149] Obtaining a fourth eigenvector of the second inverse matrix;

[0150] The second inverse matrix, the fourth eigenvector and the third submatrix are multiplied to obtain a residual matrix.

[0151] In an optional embodiment, the dimensional information includes first dimensional information to be marginalized and second dimensional information to be retained, and the Hessian matrix acquisition module 802 is specifically used to:

[0152] The first dimensional information and the second dimensional information are used to perform marginalization processing on the original Hessian matrix to construct a target Hessian matrix.

[0153] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0154] In addition, an embodiment of the present invention further provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, each process of the above-mentioned target object positioning method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0155] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned method for locating a target object is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0156] Fig. 9 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0157] The electronic device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, a processor 910, and a power supply 911. Those skilled in the art will appreciate that Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, and a pedometer.

[0158] It should be understood that in the embodiment of the present invention, the radio frequency unit 901 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving downlink data from the base station, it is sent to the processor 910 for processing; in addition, uplink data is sent to the base station. Generally, the radio frequency unit 901 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 901 can also communicate with the network and other devices through a wireless communication system.

[0159] The electronic device provides users with wireless broadband Internet access through the network module 902, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0160] The audio output unit 903 can convert the audio data received by the RF unit 901 or the network module 902 or stored in the memory 909 into an audio signal and output it as sound. Moreover, the audio output unit 903 can also provide audio output related to a specific function performed by the electronic device 900 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 903 includes a speaker, a buzzer, a receiver, etc.

[0161] The input unit 904 is used to receive audio or video signals. The input unit 904 may include a graphics processor (GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 906. The image frame processed by the graphics processor 9041 can be stored in the memory 909 (or other storage medium) or sent via the radio frequency unit 901 or the network module 902. The microphone 9042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 901 in the case of a telephone call mode.

[0162] The electronic device 900 also includes at least one sensor 905, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 9061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 9061 and / or the backlight when the electronic device 900 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 905 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0163] The display unit 906 is used to display information input by the user or information provided to the user. The display unit 906 may include a display panel 9061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0164] The user input unit 907 can be used to receive input digital or character information, and to generate signal input related to user settings and function control of the electronic device. Specifically, the user input unit 907 includes a touch panel 9071 and other input devices 9072. The touch panel 9071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 9071 or near the touch panel 9071 using any suitable object or accessory such as a finger, stylus, etc.). The touch panel 9071 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 910, receives the command sent by the processor 910 and executes it. In addition, the touch panel 9071 can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch panel 9071, the user input unit 907 may also include other input devices 9072. Specifically, other input devices 9072 may include but are not limited to a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0165] Furthermore, the touch panel 9071 may be overlaid on the display panel 9061. When the touch panel 9071 detects a touch operation on or near it, it transmits the information to the processor 910 to determine the type of the touch event. Then, the processor 910 provides a corresponding visual output on the display panel 9061 according to the type of the touch event. Fig. 9 In the figure, the touch panel 9071 and the display panel 9061 are used as two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 9071 and the display panel 9061 can be integrated to realize the input and output functions of the electronic device, which is not limited here.

[0166] The interface unit 908 is an interface for connecting an external device to the electronic device 900. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 908 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 900 or may be used to transmit data between the electronic device 900 and an external device.

[0167] The memory 909 can be used to store software programs and various data. The memory 909 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 909 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0168] The processor 910 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 909, and calling data stored in the memory 909, so as to monitor the electronic device as a whole. The processor 910 may include one or more processing units; preferably, the processor 910 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 910.

[0169] The electronic device 900 may also include a power supply 911 (such as a battery) for supplying power to each component. Preferably, the power supply 911 may be logically connected to the processor 910 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0170] In addition, the electronic device 900 includes some functional modules not shown, which will not be described in detail here.

[0171] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0172] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0173] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0174] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0175] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0176] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0177] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0179] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0180] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for locating a target object, characterized in that: include: Obtain environmental data and motion data collected by the target object; Obtaining an original Hessian matrix and dimension information of the original Hessian matrix, and performing marginalization processing on the original Hessian matrix according to the dimension information to obtain a target Hessian matrix; Solving the target Hessian matrix to obtain several sub-matrices; Performing accelerated calculation according to the plurality of sub-matrices to obtain a Jacobian matrix and a residual matrix, wherein the Jacobian matrix is ​​a matrix for representing state information of the visual inertial fusion system, and the residual matrix is ​​a matrix for optimizing variable parameters; Calculating the environment data and the motion data according to the Jacobian matrix and the residual matrix to obtain positioning information of the target object; The submatrices include at least a first submatrix, a second submatrix, and a third submatrix, and the accelerated calculation is performed according to the submatrices to obtain a Jacobian matrix and a residual matrix, including: Adding the first submatrix and the transposed first submatrix to obtain a first target matrix; Perform a matrix operation according to the first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix; Multiplying the first inverse matrix with the second submatrix and the third submatrix respectively to obtain a second target matrix corresponding to the second submatrix and a third target matrix corresponding to the third submatrix; Performing a matrix operation according to the second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix; Performing matrix operations using the third eigenvalue information of the diagonal matrix to obtain a Jacobian matrix; A matrix operation is performed using the second inverse matrix and the third submatrix to obtain a residual matrix.

2. The method according to claim 1, characterized in that The performing a matrix operation according to the first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix includes: Obtaining a first eigenvalue and a first eigenvector of the first target matrix; A first matrix constructed by the first eigenvalues ​​and a second matrix constructed by the first eigenvectors are multiplied to obtain a first inverse matrix corresponding to the first target matrix.

3. The method according to claim 1, characterized in that The performing matrix operation according to the second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix includes: Obtaining a second eigenvalue and a second eigenvector of the second target matrix; Multiplying a third matrix constructed by the second eigenvalues ​​and a fourth matrix constructed by the second eigenvectors to obtain a target vector and a target inverse vector corresponding to the second target matrix; A square root operation is performed on each scalar in the target vector to obtain a diagonal matrix corresponding to the target vector, and a square root operation is performed on each scalar in the target inverse vector to obtain a second inverse matrix corresponding to the diagonal matrix.

4. The method according to claim 1, characterized in that: The step of performing matrix operation using the third characteristic information of the diagonal matrix to obtain a Jacobian matrix includes: Obtaining a third eigenvector of the diagonal matrix; The diagonal matrix and the third eigenvector are multiplied to obtain a Jacobian matrix.

5. The method according to claim 1, characterized in that The step of performing a matrix operation on the second inverse matrix and the third submatrix to obtain a residual matrix includes: Obtaining a fourth eigenvector of the second inverse matrix; The second inverse matrix, the fourth eigenvector and the third submatrix are multiplied to obtain a residual matrix.

6. The method according to claim 1, characterized in that The dimensional information includes first dimensional information to be marginalized and second dimensional information to be retained, and performing marginalization processing on the original Hessian matrix according to the dimensional information to obtain a target Hessian matrix includes: The first dimensional information and the second dimensional information are used to perform marginalization processing on the original Hessian matrix to construct a target Hessian matrix.

7. A visual inertial fusion system, characterized in that: The visual inertial fusion system includes a visual and inertial measurement unit, a control instruction memory, a tensor data memory, a tensor arithmetic logic operation array, and a tensor acceleration calculation control unit; the tensor data memory stores the environmental data and motion data collected by the visual and inertial measurement unit; wherein, The tensor acceleration calculation control unit is used to transmit the positioning operation instruction to the tensor arithmetic logic operation array in response to receiving the positioning operation instruction transmitted by the control instruction memory; The tensor arithmetic logic operation array is used to obtain an original Hessian matrix and dimension information for the original Hessian matrix from the tensor data memory according to the positioning operation instruction, and perform marginalization processing on the original Hessian matrix according to the dimension information to obtain a target Hessian matrix; solve the target Hessian matrix to obtain a plurality of sub-matrices; perform accelerated calculation according to the plurality of sub-matrices to obtain a Jacobian matrix and a residual matrix; calculate the environmental data and the motion data according to the Jacobian matrix and the residual matrix to obtain the positioning information of the target object, wherein the Jacobian matrix is ​​a matrix used to characterize the state information of the visual inertial fusion system, and the residual matrix is ​​a matrix used to optimize variable parameters; The tensor data memory is used to store the target Hessian matrix, the Jacobian matrix, the residual matrix and the positioning information; The tensor arithmetic logic operation array includes a calculation instruction parsing unit, a plurality of arithmetic logic units and an adder tree; The computing instruction parsing unit is used to parse the positioning operation instruction, obtain the environmental data, the motion data and the original Hessian matrix from the tensor data memory, and obtain dimension information for the original Hessian matrix; The arithmetic logic unit, wherein the submatrix includes at least a first submatrix, a second submatrix and a third submatrix; is used to add the first submatrix and the transposed first submatrix to obtain a first target matrix; perform matrix operations according to the first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix; multiply the first inverse matrix with the second submatrix and the third submatrix respectively to obtain a second target matrix corresponding to the second submatrix and a third target matrix corresponding to the third submatrix; perform matrix operations according to the second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix; perform matrix operations using the third characteristic information of the diagonal matrix to obtain a Jacobian matrix; perform matrix operations using the second inverse matrix and the third submatrix to obtain a residual matrix; The adder tree is used to accelerate operations when performing matrix multiplication.

8. The system according to claim 7, characterized in that The arithmetic logic unit includes a first arithmetic logic unit and a plurality of second arithmetic logic units; wherein the first arithmetic logic unit is used to perform at least one operation of addition, subtraction, multiplication, division and square root, and the second arithmetic logic unit is used to perform at least one operation of addition, subtraction and multiplication.

9. A positioning device for a target object, characterized in that: include: A data acquisition module is used to acquire environmental data and motion data collected by the target object; A Hessian matrix acquisition module, used to obtain an original Hessian matrix and dimension information of the original Hessian matrix, and perform marginalization processing on the original Hessian matrix according to the dimension information to obtain a target Hessian matrix; A solution module, used for solving the target Hessian matrix to obtain a plurality of sub-matrices; An accelerated calculation module, used for performing accelerated calculation according to the plurality of sub-matrices to obtain a Jacobian matrix and a residual matrix, wherein the Jacobian matrix is ​​a matrix for representing state information of the visual inertial fusion system, and the residual matrix is ​​a matrix for optimizing variable parameters; A positioning module, used to calculate the environmental data and the motion data according to the Jacobian matrix and the residual matrix to obtain the positioning information of the target object; The accelerated computing module is further specifically used for: The sub-matrix at least includes a first sub-matrix, a second sub-matrix and a third sub-matrix; Adding the first submatrix and the transposed first submatrix to obtain a first target matrix; Perform a matrix operation according to the first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix; Multiplying the first inverse matrix with the second submatrix and the third submatrix respectively to obtain a second target matrix corresponding to the second submatrix and a third target matrix corresponding to the third submatrix; Matrix operations are performed according to the second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix; matrix operations are performed using the third characteristic information of the diagonal matrix to obtain a Jacobian matrix; matrix operations are performed using the second inverse matrix and the third submatrix to obtain a residual matrix.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 6 when executing the program stored in the memory.

11. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 6.

Images