Positioning method, device and equipment based on vision and IMU (Inertial Measurement Unit) and storage medium

Abstract

The invention discloses a positioning method, device and equipment based on vision and IMU (Inertial Measurement Unit) and a computer readable storage medium, and relates to the technical field of synchronous positioning and mapping, the method comprises the following steps: dynamically calculating a first pose according to a preset image input frame rate according to an input image and IMU data; a newly calculated first pose is dynamically obtained according to a preset pose output frame rate, accumulated integral data of a target IMU data set is calculated, the target IMU data set comprises IMU data input from the input moment of a target image to the current moment, and the target image is the latest frame image adopted for calculating the newly calculated first pose; calculating a second pose according to the newly calculated first pose and the accumulated integral data; and outputting the first pose and the second pose. The invention provides a positioning scheme based on vision and IMU (Inertial Measurement Unit), and the pose output frame rate is improved.

Description

Technical Field

[0001] The present invention relates to the field of synchronous positioning and mapping technology, and in particular to a positioning method, device, equipment and computer-readable storage medium based on vision and IMU (Inertial Measurement Unit). Background Art

[0002] SLAM (Simultaneous Localization and Mapping) is a technology that uses various sensors to locate its own position and posture. It is currently widely used, for example in VR (Virtual Reality) devices, AR (Augmented Reality) devices, sweeping robots, etc. Visual SLAM is a branch of SLAM that mainly relies on cameras for positioning. SLAM supports the pure vision VO (visual odometry) mode and the vision + IMU prediction VIO (Visual-Inertial Odometry) mode. When the VIO mode outputs posture based on vision + IMU, it is based on the image input frame rate of the camera, that is, one frame of posture is calculated for each frame of image input, resulting in a low posture output frame rate and serious lag problems during device use. Summary of the invention

[0003] The main purpose of the present invention is to provide a positioning method, device, equipment and computer-readable storage medium based on vision and IMU, aiming to provide a positioning solution based on vision and IMU to improve the output frame rate of posture.

[0004] To achieve the above object, the present invention provides a positioning method based on vision and IMU, the method comprising the following steps:

[0005] Dynamically calculate the first pose based on the input image and IMU data at the preset image input frame rate;

[0006] Dynamically obtain the latest calculated first pose according to a preset pose output frame rate, and calculate the cumulative integral data of the target IMU data group, wherein the target IMU data group includes the IMU data input from the input time of the target image to the current time, and the target image is the latest frame image used to calculate the latest calculated first pose;

[0007] Calculate a second posture according to the latest calculated first posture and the accumulated integral data;

[0008] The first pose and the second pose are output.

[0009] Optionally, the step of calculating the cumulative integral data of the target IMU data group includes:

[0010] If the latest calculated first position currently obtained is different from the first position obtained last time, clear the historical integral intermediate data, obtain the target IMU data group, calculate the cumulative integral data according to the target IMU data group, and use the cumulative integral data as new integral intermediate data;

[0011] If the latest calculated first position currently obtained is the same as the first position obtained last time, then obtain the latest IMU data input since the last time the first position was obtained, calculate the cumulative integral data based on the latest IMU data and the historical integral intermediate data, and use the cumulative integral data to update the historical integral intermediate data.

[0012] Optionally, the step of dynamically calculating the first pose according to the input image and IMU data at a preset image input frame rate includes:

[0013] Receive input IMU data and cache it in the IMU data array;

[0014] After each frame of image is input, the IMU data array is traversed, and the IMU data in the IMU data array whose input time is between the input time of the current frame of image and the input time of the previous frame of image is retained, and the remaining IMU data is removed;

[0015] After removing the IMU data, the first pose is calculated based on the current frame image and the IMU data retained in the IMU data array.

[0016] Optionally, the step of calculating the second posture according to the latest calculated first posture and the accumulated integral data comprises:

[0017] The position in the first posture is corrected by using the displacement in the accumulated integral data, and the posture in the first posture is corrected by using the rotation angle in the accumulated integral data, and a second posture is obtained according to the corrected position and posture.

[0018] Optionally, the step of calculating the cumulative integral data of the target IMU data group includes:

[0019] The displacement is obtained by integrating the acceleration data in the target IMU data set;

[0020] The rotation angle is obtained by integrating the angular velocity data in the target IMU data group;

[0021] The displacement and the rotation angle are used as cumulative integration data.

[0022] Optionally, before the step of dynamically acquiring the latest calculated first pose according to the preset pose output frame rate and calculating the cumulative integral data of the target IMU data group, it also includes:

[0023] Receive a request to set the pose output frame rate;

[0024] If the target output frame rate indicated in the pose output frame rate setting request is less than or equal to the preset image input frame rate, outputting the first pose calculated dynamically;

[0025] If the target output frame rate is greater than the preset image input frame rate, the preset posture output frame rate is set to be greater than or equal to the target output frame rate, and the steps of dynamically obtaining the latest calculated first posture according to the preset posture output frame rate and calculating the cumulative integral data of the target IMU data group are executed.

[0026] Optionally, the preset posture output frame rate is the same as the input frame rate of the IMU data.

[0027] To achieve the above object, the present invention further provides a positioning device based on vision and IMU, the device comprising:

[0028] The first pose calculation module is used to dynamically calculate the first pose according to the input image and IMU data at a preset image input frame rate;

[0029] An integration module, used to dynamically obtain the latest calculated first pose according to a preset pose output frame rate, and calculate the cumulative integral data of the target IMU data group, wherein the target IMU data group includes the IMU data input from the input time of the target image to the current time, and the target image is the latest frame image used to calculate the latest calculated first pose;

[0030] A second posture calculation module, used for calculating a second posture according to the latest calculated first posture and the accumulated integral data;

[0031] An output module is used to output the first posture and the second posture.

[0032] To achieve the above-mentioned objectives, the present invention also provides a vision and IMU-based positioning device, which includes: a memory, a processor, and a vision and IMU-based positioning program stored in the memory and executable on the processor, wherein the vision and IMU-based positioning program, when executed by the processor, implements the steps of the vision and IMU-based positioning method as described above.

[0033] In addition, to achieve the above-mentioned objectives, the present invention also proposes a computer-readable storage medium, on which a vision and IMU-based positioning program is stored. When the vision and IMU-based positioning program is executed by a processor, the steps of the vision and IMU-based positioning method as described above are implemented.

[0034] In an embodiment of the present invention, first, the first pose is dynamically calculated according to the input image and IMU data according to a preset image input frame rate, so as to realize conventional pose calculation based on the image input frame rate; then, the latest calculated first pose is dynamically obtained according to the preset pose output frame rate, and the cumulative integral data of the target IMU data group is calculated, the target IMU data group includes the IMU data input from the input moment of the target image to the current moment, the target image is the latest frame image used to calculate the latest calculated first pose, and provides a data basis for calculating the second pose; finally, the second pose is calculated according to the latest calculated first pose and the cumulative integral data, so as to realize the calculation of the second pose by using the cumulative integral data of the IMU data in combination with the first pose calculated according to the image input frame rate, so as to compensate the output gap of the first pose by the second pose, thereby improving the pose output frame rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of an embodiment of a positioning method based on vision and IMU of the present invention;

[0036] Figure 2 A timing diagram of a posture calculation process involved in an embodiment of the present invention;

[0037] Figure 3 A first pose calculation flow chart involved in an embodiment of the present invention;

[0038] Figure 4 A second posture calculation flow chart involved in an embodiment of the present invention;

[0039] Figure 5 It is a functional module schematic diagram of a preferred embodiment of a positioning device based on vision and IMU of the present invention;

[0040] Figure 6 It is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention.

[0041] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0042] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0043] Reference Figure 1, Figure 1 The figure is a flow chart of an embodiment of a positioning method based on vision and IMU of the present invention.

[0044] The embodiments of the present invention provide embodiments of the positioning method based on vision and IMU. It should be noted that although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that here. In this embodiment, the execution subject of the positioning method based on vision and IMU can be an AR device, a VR device, a sweeping robot, a smart phone, a personal computer and other devices, which are not limited in this embodiment. In this embodiment, for ease of expression, the following embodiments are described with the positioning system as the execution subject. In this embodiment, the positioning method based on vision and IMU includes the following steps S10 to S40:

[0045] Step S10, dynamically calculating the first pose according to the input image and IMU data at a preset image input frame rate.

[0046] The preset image input frame rate refers to the image input frame rate at which the camera inputs the image to the positioning system after collecting the image, which is pre-set in the positioning system, for example, 30Hz. IMU data is collected by the inertial measurement unit and input to the positioning system for positioning. The input frame rate of the IMU data is generally faster than the preset image input frame rate, for example, 1000Hz. The positioning system receives images and IMU data, and dynamically calculates the first pose according to the preset image input frame rate, that is, each time a frame of image is received, the pose is calculated once based on the currently received image and IMU data (hereinafter referred to as the first pose, to distinguish the pose calculated by another method in this embodiment). The method for calculating the first pose based on the input image and IMU data can refer to the calculation method in the relevant technology, and is not limited here.

[0047] In a feasible implementation manner, the step S10 includes S101 to S103:

[0048] Step S101, receiving input IMU data and caching it in an IMU data array.

[0049] The positioning system can receive IMU data input by the inertial measurement unit and buffer it in an array (hereinafter referred to as IMU data array).

[0050] Step S102, after each frame of image is input, the IMU data array is traversed, and the IMU data in the IMU data array whose input time is between the input time of the current frame of image and the input time of the previous frame of image is retained, and the remaining IMU data is removed.

[0051] After each frame of image is input (input refers to the image captured by the camera and input into the positioning system), the positioning system can traverse the IMU data array and retain the IMU data in the IMU data array whose input time is between the input image (called the current frame image) and the input time of the previous frame image, and remove the rest of the IMU data. Figure 2 As shown in the figure, a schematic diagram of the calculation process of the first pose and the second pose in time sequence is given as an example, in which C1, C2 and C3 represent the input frames, P VIO 1. P VIO 2. P VIO 3... represents the calculated first pose of each frame, the part covered by the gray box represents the time of calculating the first pose, I1, I2... represent the input IMU data of each frame, P imu 11. P imu 12. P imu 13... represents the calculated second pose of each frame. Assuming that the current frame image is C2, it is equivalent to retaining the IMU data input between the two frames of images C1 and C2, that is, I1-I7.

[0052] Step S103, after removing the IMU data, the first position is calculated based on the current frame image and the IMU data retained in the IMU data array.

[0053] After removing the IMU data that needs to be removed from the IMU data array, the positioning system can calculate the first pose based on the current frame image and the IMU data retained in the IMU data array. The specific method of calculating the first pose is not limited in this embodiment.

[0054] For example, in one feasible implementation, the first pose can be calculated according to the process shown in 3.

[0055] Step S20, dynamically acquiring the latest calculated first pose according to a preset pose output frame rate, and calculating the cumulative integral data of the target IMU data group, wherein the target IMU data group includes the IMU data input from the input moment of the target image to the current moment, and the target image is the latest frame image used to calculate the latest calculated first pose.

[0056] The preset posture output frame rate can be a posture output frame rate that is set in advance according to needs. When the positioning system is applied to different devices, the posture output frame rate that needs to be achieved is different. Therefore, in this embodiment, there is no restriction on the preset posture output frame rate. The preset posture output frame rate can be set to be larger than the preset image input frame rate, and the preset posture output frame rate is the posture output frame rate that can be achieved by the positioning method in this embodiment. Therefore, the positioning method in this embodiment can improve the posture output frame rate compared to the current positioning method.

[0057] The preset posture output frame rate is the frequency of the output posture. The positioning system dynamically obtains the latest calculated first posture and calculates the cumulative integral data of the target IMU data group according to this frequency. Each time the latest calculated first posture and the corresponding cumulative integral data are obtained, a second posture is calculated based on the obtained first posture and the corresponding cumulative integral data, thereby achieving the effect that the posture output frame rate can reach the preset posture output frame rate.

[0058] It is understandable that, since the positioning system dynamically calculates the first pose according to the preset image input frame rate, new first poses are continuously generated as the image is input, and the positioning system dynamically obtains the latest calculated first pose according to the preset pose output frame rate. The calculation of the first pose may use at least one frame of image, and the latest acquired frame of image is called the target image for distinction, and the IMU data input from the input moment of the target image to the current moment (the moment of obtaining the latest calculated first pose) is called the target IMU data group for distinction. It is understandable that, since the input frame rate of IMU data is generally greater than the input frame rate of the image, and the preset pose output frame rate is set to be greater than the image input frame rate, the target IMU data group includes multiple frames of IMU data.

[0059] Calculating the cumulative score of the target IMU data group is to integrate each frame of IMU data in the IMU data group. The specific integration process is not limited in this embodiment.

[0060] In a feasible implementation, the preset pose output frame rate can be set to be the same as the input frame rate of the IMU data, that is, assuming that the input frame rate of the IMU data is 1000HZ, the preset pose output frame rate can be set to 1000Hz. Then, after receiving each frame of IMU data, the positioning system obtains the latest calculated first pose and the corresponding accumulated integral data, and calculates a second pose.

[0061] For example, Figure 2 As shown in the figure, the preset pose output frame rate is the same as the input frame rate of the IMU data. Assuming that the current moment is after the IMU data I5 is input, the latest first pose obtained is P VIO 1, the corresponding target image is C1, the target IMU data array is [I1, I2, I3, I4, I5], the cumulative integral data is calculated based on the target iMU data array, and then the cumulative integral data and P are used to calculate the cumulative integral data. VIO 1 Calculate the second pose P imu 11. It is understandable, then, that P imu 12 is based on P VIO 1 and [I1, I2, I3, I4, I6] calculated.

[0062] Step S30, calculating a second posture according to the most recently calculated first posture and the accumulated integral data.

[0063] The positioning system calculates a posture (called the second posture to distinguish the first posture) based on the latest calculated first posture and the accumulated integral data. The principle of calculating the second posture based on the latest calculated first posture and the accumulated integral data is that the integral of multiple frames of IMU data can reflect the offset of the positioned position and posture, and the offset can be used to correct the first posture, thereby making the first posture more accurate, and at the same time, it can supplement the posture missing in the image input gap, thereby improving the posture output frame rate. There are many ways to calculate the second posture based on the latest calculated first posture and the accumulated integral data, which are not limited in this embodiment.

[0064] Step S40: output the first posture and the second posture.

[0065] Although the steps are numbered in the embodiment, the numbering does not limit the execution order of the steps, and the timing of outputting the first posture and the second posture is not limited in the present embodiment. In a specific implementation, the first posture can be output in real time or in non-real time as needed. In the case of real-time output, the positioning system outputs the posture after each posture (the first posture or the second posture) is calculated, that is, step S40 does not mean outputting the first posture after the second posture is calculated. In the case of non-real-time output, the positioning system can cache or store each posture (the first posture or the second posture) after calculating it, and output it when it is needed. The output can be, for example, output to a back-end processing module that needs to use the posture information, and the object output in the specific application scenario can be set as needed.

[0066] In one possible implementation, referring to Figure 4 The step of calculating the cumulative integral data of the target IMU data group in step S20 includes S201-S202:

[0067] Step S201, if the latest calculated first position currently obtained is different from the first position obtained last time, clear the historical integral intermediate data, obtain the target IMU data group, calculate the cumulative integral data based on the target IMU data group, and use the cumulative integral data as new integral intermediate data.

[0068] If the latest calculated first pose currently acquired is different from the first pose acquired last time, it means that a new first pose is generated, which means that the target image corresponding to the first pose acquired this time is different from the target image corresponding to the first pose acquired last time. Furthermore, the target IMU data group corresponding to the first pose acquired this time is also different from the target IMU data group corresponding to the first pose acquired last time. In this case, the historical integral intermediate data can be used to obtain the target IMU data group corresponding to the first pose acquired this time from the received IMU data. Calculate the cumulative integral data based on the acquired target IMU data group. The calculated cumulative integral data can be used as new integral intermediate data for use in calculating the second pose of the next frame.

[0069] Step S202, if the latest calculated first position currently obtained is the same as the first position obtained last time, then obtain the latest IMU data input since the first position was last obtained, calculate the cumulative integral data based on the latest IMU data and the historical integral intermediate data, and use the cumulative integral data to update the historical integral intermediate data.

[0070] If the latest calculated first pose currently obtained is different from the first pose obtained last time, it means that no new first pose is generated, then it means that the target image corresponding to the first pose obtained this time is the same as the target image corresponding to the first pose obtained last time. Furthermore, the target IMU data group corresponding to the first pose obtained this time has one or more newly received frames of IMU data compared to the target IMU data group corresponding to the first pose obtained last time. Therefore, the integral intermediate data calculated when the second pose was calculated last time can be used, combined with the newly received IMU data, to calculate the cumulative integral data of the target IMU data group corresponding to the first pose obtained this time, thereby simplifying the calculation process, improving the calculation efficiency, and ultimately improving the pose output frame rate.

[0071] In this embodiment, first, the first pose is dynamically calculated according to the input image and IMU data according to the preset image input frame rate, so as to realize the conventional pose calculation based on the image input frame rate; then, the latest calculated first pose is dynamically obtained according to the preset pose output frame rate, and the cumulative integral data of the target IMU data group is calculated, the target IMU data group includes the IMU data input from the input time of the target image to the current time, the target image is the latest frame image used to calculate the latest calculated first pose, and provides a data basis for calculating the second pose; finally, the second pose is calculated according to the latest calculated first pose and the cumulative integral data, so as to realize the calculation of the second pose by using the cumulative integral data of the IMU data combined with the first pose calculated according to the image input frame rate, so as to compensate the output gap of the first pose by the second pose, so as to improve the pose output frame rate. For example, when the image input frame rate is 30HZ and the IMU data input frame rate is 1000HZ, the pose output frame rate can be increased from 30HZ to 1000HZ.

[0072] Based on the above first embodiment, a second embodiment of the prompting method of the VR device of the present invention is proposed. In this embodiment, the step S30 includes S301:

[0073] Step S301, using the displacement in the accumulated integral data to correct the position in the first posture, and using the rotation angle in the accumulated integral data to correct the posture in the first posture, and obtaining a second posture according to the corrected position and posture.

[0074] In this embodiment, a feasible implementation method for calculating the second posture based on the first posture and the accumulated integral data is proposed. The accumulated integral data includes displacement and rotation angle, and the first posture includes position and attitude. The positioning system can use the displacement to correct the position in the first posture, and use the rotation angle to correct the attitude in the first posture, so as to obtain the second posture according to the corrected position and attitude. In a feasible implementation method, the displacement includes displacement in three axes in the spatial coordinate system, and the displacement in each axis can be used to respectively correct the position coordinates in each axis in the first posture; the rotation angle includes the rotation angle around the three axes in the spatial coordinate system, and each rotation angle can be used to respectively correct the attitude angle in each axis in the first posture.

[0075] In one feasible implementation, the step of calculating the cumulative integral data of the target IMU data group in step S20 includes S203 to S205:

[0076] Step S203, obtaining displacement by integrating the acceleration data in the target IMU data group.

[0077] In this embodiment, a feasible implementation method for calculating the cumulative integral data of the target IMU data group is proposed. The target IMU data group includes acceleration data, and the displacement can be obtained by integrating the acceleration data. In a feasible implementation method, the acceleration data includes acceleration data on three axes in the spatial coordinate system, and the acceleration data on the three axes can be integrated separately to obtain the velocities on the three axes, and then the velocities on the three axes are integrated separately to obtain the displacements on the three axes.

[0078] Step S204, obtaining the rotation angle by integrating the angular velocity data in the target IMU data group.

[0079] The target IMU data set includes angular velocity data, and the rotation angle can be obtained by integrating the angular velocity data. In one feasible implementation, the angular velocity data includes angular velocities rotating around three axes in the spatial coordinate system, and the angular velocities rotating around the three axes can be integrated separately to obtain the rotation angles around the three axes.

[0080] Step S205: taking the displacement and the rotation angle as cumulative integral data.

[0081] After obtaining the displacement and rotation angle, the positioning system uses the displacement and rotation angle as cumulative integral data, and then calculates the second posture based on the cumulative integral data and the first posture.

[0082] In a feasible implementation manner, before step S20, the following steps are further included: S50 to S70:

[0083] Step S50, receiving a pose output frame rate setting request.

[0084] The pose output frame rate setting request may be a request sent by other modules in the device to the positioning system, and the sending subject is not limited in this embodiment. The pose output frame rate setting request indicates the required pose output frame rate, which is hereinafter referred to as the target output frame rate for distinction.

[0085] Step S60: If the target output frame rate indicated in the pose output frame rate setting request is less than or equal to the preset image input frame rate, the dynamically calculated first pose is output.

[0086] If the target output frame rate is less than or equal to the preset image input frame rate, it means that the required output pose frame rate in the actual application scenario does not exceed the input frame rate of the image. In this case, the first pose can be output after it is calculated, without the need to calculate the second pose. This saves device computing resources while meeting the actual scenario requirements.

[0087] Step S70, if the target output frame rate is greater than the preset image input frame rate, the preset posture output frame rate is set to be greater than or equal to the target output frame rate, and the steps of dynamically obtaining the latest calculated first posture according to the preset posture output frame rate and calculating the cumulative integral data of the target IMU data group are executed.

[0088] If the target output frame rate is greater than the preset image input frame rate, it means that the pose frame rate required to be output in the actual application scenario is greater than the input frame rate of the image. In order to meet the needs of the actual application scenario and avoid lag, the preset pose output frame rate can be set to be greater than or equal to the target output frame rate, and steps S20 to S40 are executed according to the preset pose output frame rate to calculate the second pose to make up for the lack of frame rate, thereby achieving the effect that the actual output frame rate can meet the expected output frame rate.

[0089] In addition, the embodiment of the present invention also proposes a positioning device based on vision and IMU, referring to Figure 5 , the device comprises:

[0090] A first pose calculation module 10 is used to dynamically calculate the first pose according to the input image and IMU data at a preset image input frame rate;

[0091] An integration module 20, used to dynamically obtain the latest calculated first pose according to a preset pose output frame rate, and calculate the cumulative integral data of the target IMU data group, wherein the target IMU data group includes the IMU data input from the input time of the target image to the current time, and the target image is the latest frame image used to calculate the latest calculated first pose;

[0092] A second posture calculation module 30, used for calculating a second posture according to the latest calculated first posture and the accumulated integral data;

[0093] The output module 40 is configured to output the first posture and the second posture.

[0094] In one feasible implementation, the integration module 20 is further used for:

[0095] If the latest calculated first position currently obtained is different from the first position obtained last time, clear the historical integral intermediate data, obtain the target IMU data group, calculate the cumulative integral data according to the target IMU data group, and use the cumulative integral data as new integral intermediate data;

[0096] If the latest calculated first position currently obtained is the same as the first position obtained last time, then obtain the latest IMU data input since the last time the first position was obtained, calculate the cumulative integral data based on the latest IMU data and the historical integral intermediate data, and use the cumulative integral data to update the historical integral intermediate data.

[0097] In one feasible implementation, the first posture calculation module 10 is further used for:

[0098] Receive input IMU data and cache it in the IMU data array;

[0099] After each frame of image is input, the IMU data array is traversed, and the IMU data in the IMU data array whose input time is between the input time of the current frame of image and the input time of the previous frame of image is retained, and the remaining IMU data is removed;

[0100] After removing the IMU data, the first pose is calculated based on the current frame image and the IMU data retained in the IMU data array.

[0101] In one feasible implementation, the second posture calculation module 30 is further used for:

[0102] The position in the first posture is corrected by using the displacement in the accumulated integral data, and the posture in the first posture is corrected by using the rotation angle in the accumulated integral data, and a second posture is obtained according to the corrected position and posture.

[0103] In one feasible implementation, the integration module 20 is further used for:

[0104] The displacement is obtained by integrating the acceleration data in the target IMU data set;

[0105] The rotation angle is obtained by integrating the angular velocity data in the target IMU data group;

[0106] The displacement and the rotation angle are used as cumulative integration data.

[0107] In one feasible implementation, the vision and IMU-based positioning device further includes:

[0108] A receiving module, used for receiving a request for setting a posture output frame rate;

[0109] The output module 40 is further configured to output the first posture calculated dynamically if the target output frame rate indicated in the posture output frame rate setting request is less than or equal to the preset image input frame rate;

[0110] The integration module 20 is also used to set the preset posture output frame rate to be greater than or equal to the target output frame rate if the target output frame rate is greater than the preset image input frame rate, and to perform the operations of dynamically acquiring the latest calculated first posture according to the preset posture output frame rate, and calculating the cumulative integral data of the target IMU data group.

[0111] In one feasible implementation, the preset posture output frame rate is the same as the input frame rate of the IMU data.

[0112] In addition, the embodiment of the present invention also proposes a positioning device based on vision and IMU, such as Figure 6 As shown, Figure 6 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention. It should be noted that the positioning device based on vision and IMU in the embodiment of the present invention can be an AR device, a VR device, a sweeping robot, a smart phone, a personal computer and other devices, and no specific restrictions are made here.

[0113] like Figure 6 As shown, the vision and IMU-based positioning device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0114] Those skilled in the art will understand that Figure 6 The device structure shown in the figure does not constitute a limitation on the positioning device based on vision and IMU, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0115] like Figure 6 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a positioning program based on vision and IMU. The operating system is a program that manages and controls the hardware and software resources of the device, and supports the operation of the positioning program based on vision and IMU and other software or programs. Figure 6In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 can be used to call the vision and IMU-based positioning program stored in the memory 1005 and perform the following operations:

[0116] Dynamically calculate the first pose based on the input image and IMU data at the preset image input frame rate;

[0117] Dynamically obtain the latest calculated first pose according to a preset pose output frame rate, and calculate the cumulative integral data of the target IMU data group, wherein the target IMU data group includes the IMU data input from the input time of the target image to the current time, and the target image is the latest frame image used to calculate the latest calculated first pose;

[0118] Calculate a second posture according to the latest calculated first posture and the accumulated integral data;

[0119] The first pose and the second pose are output.

[0120] In one feasible implementation, the operation of calculating the cumulative integral data of the target IMU data group includes:

[0121] If the latest calculated first position currently obtained is different from the first position obtained last time, clear the historical integral intermediate data, obtain the target IMU data group, calculate the cumulative integral data according to the target IMU data group, and use the cumulative integral data as new integral intermediate data;

[0122] If the latest calculated first position currently obtained is the same as the first position obtained last time, then obtain the latest IMU data input since the last time the first position was obtained, calculate the cumulative integral data based on the latest IMU data and the historical integral intermediate data, and use the cumulative integral data to update the historical integral intermediate data.

[0123] In a feasible implementation manner, the operation of dynamically calculating the first pose according to the input image and IMU data at a preset image input frame rate includes:

[0124] Receive input IMU data and cache it in the IMU data array;

[0125] After each frame of image is input, the IMU data array is traversed, and the IMU data in the IMU data array whose input time is between the input time of the current frame of image and the input time of the previous frame of image is retained, and the remaining IMU data is removed;

[0126] After removing the IMU data, the first pose is calculated based on the current frame image and the IMU data retained in the IMU data array.

[0127] In one feasible implementation, the operation of calculating the second posture according to the latest calculated first posture and the accumulated integral data includes:

[0128] The position in the first posture is corrected by using the displacement in the accumulated integral data, and the posture in the first posture is corrected by using the rotation angle in the accumulated integral data, and a second posture is obtained according to the corrected position and posture.

[0129] In one feasible implementation, the operation of calculating the cumulative integral data of the target IMU data group includes:

[0130] The displacement is obtained by integrating the acceleration data in the target IMU data set;

[0131] The rotation angle is obtained by integrating the angular velocity data in the target IMU data group;

[0132] The displacement and the rotation angle are used as cumulative integration data.

[0133] In a feasible implementation manner, before the operation of dynamically acquiring the latest calculated first pose according to the preset pose output frame rate and calculating the cumulative integral data of the target IMU data group, it also includes:

[0134] Receive a request to set the pose output frame rate;

[0135] If the target output frame rate indicated in the pose output frame rate setting request is less than or equal to the preset image input frame rate, outputting the first pose calculated dynamically;

[0136] If the target output frame rate is greater than the preset image input frame rate, the preset posture output frame rate is set to be greater than or equal to the target output frame rate, and the operations of dynamically obtaining the latest calculated first posture according to the preset posture output frame rate and calculating the cumulative integral data of the target IMU data group are performed.

[0137] In one feasible implementation, the preset posture output frame rate is the same as the input frame rate of the IMU data.

[0138] In addition, an embodiment of the present invention also proposes a computer-readable storage medium, on which a vision- and IMU-based positioning program is stored. When the vision- and IMU-based positioning program is executed by a processor, the steps of the vision- and IMU-based positioning method described below are implemented.

[0139] The various embodiments of the positioning device based on vision and IMU and the computer-readable storage medium of the present invention can all refer to the various embodiments of the positioning method based on vision and IMU of the present invention, which will not be repeated here.

[0140] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0141] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0143] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A positioning method based on vision and IMU, characterized in that: The method comprises the following steps: Dynamically calculate the first pose based on the input image and IMU data at the preset image input frame rate; Dynamically obtain the latest calculated first pose according to a preset pose output frame rate, and calculate the cumulative integral data of the target IMU data group, wherein the target IMU data group includes the IMU data input from the input time of the target image to the current time, and the target image is the latest frame image used to calculate the latest calculated first pose; Calculate a second posture according to the latest calculated first posture and the accumulated integral data; The first pose and the second pose are output.

2. The positioning method based on vision and IMU as claimed in claim 1, characterized in that: The step of calculating the cumulative integral data of the target IMU data group comprises: If the latest calculated first position currently obtained is different from the first position obtained last time, clear the historical integral intermediate data, obtain the target IMU data group, calculate the cumulative integral data according to the target IMU data group, and use the cumulative integral data as new integral intermediate data; If the latest calculated first position currently obtained is the same as the first position obtained last time, then obtain the latest IMU data input since the last time the first position was obtained, calculate the cumulative integral data based on the latest IMU data and the historical integral intermediate data, and use the cumulative integral data to update the historical integral intermediate data.

3. The positioning method based on vision and IMU as claimed in claim 1, characterized in that: The step of dynamically calculating the first pose according to the input image and IMU data at a preset image input frame rate comprises: Receive input IMU data and cache it in the IMU data array; After each frame of image is input, the IMU data array is traversed, and the IMU data in the IMU data array whose input time is between the input time of the current frame of image and the input time of the previous frame of image is retained, and the remaining IMU data is removed; After removing the IMU data, the first pose is calculated based on the current frame image and the IMU data retained in the IMU data array.

4. The positioning method based on vision and IMU as claimed in claim 1, characterized in that: The step of calculating the second posture according to the latest calculated first posture and the accumulated integral data comprises: The position in the first posture is corrected by using the displacement in the accumulated integral data, and the posture in the first posture is corrected by using the rotation angle in the accumulated integral data, and a second posture is obtained according to the corrected position and posture.

5. The positioning method based on vision and IMU as claimed in claim 1, characterized in that: The step of calculating the cumulative integral data of the target IMU data group comprises: The displacement is obtained by integrating the acceleration data in the target IMU data set; The rotation angle is obtained by integrating the angular velocity data in the target IMU data group; The displacement and the rotation angle are used as cumulative integration data.

6. The positioning method based on vision and IMU as claimed in any one of claims 1 to 5, characterized in that: Before the step of dynamically acquiring the latest calculated first pose according to the preset pose output frame rate and calculating the cumulative integral data of the target IMU data group, the method further includes: Receive a request to set the pose output frame rate; If the target output frame rate indicated in the pose output frame rate setting request is less than or equal to the preset image input frame rate, outputting the first pose calculated dynamically; If the target output frame rate is greater than the preset image input frame rate, the preset posture output frame rate is set to be greater than or equal to the target output frame rate, and the steps of dynamically obtaining the latest calculated first posture according to the preset posture output frame rate and calculating the cumulative integral data of the target IMU data group are executed.

7. The positioning method based on vision and IMU as claimed in any one of claims 1 to 5, characterized in that: The preset posture output frame rate is the same as the input frame rate of the IMU data.

8. A positioning device based on vision and IMU, characterized in that: The device comprises: The first pose calculation module is used to dynamically calculate the first pose according to the input image and IMU data at a preset image input frame rate; An integration module, used to dynamically obtain the latest calculated first pose according to a preset pose output frame rate, and calculate the cumulative integral data of the target IMU data group, wherein the target IMU data group includes the IMU data input from the input time of the target image to the current time, and the target image is the latest frame image used to calculate the latest calculated first pose; A second posture calculation module, used for calculating a second posture according to the latest calculated first posture and the accumulated integral data; An output module is used to output the first posture and the second posture.

9. A positioning device based on vision and IMU, characterized in that: The vision and IMU-based positioning device includes: a memory, a processor, and a vision and IMU-based positioning program stored in the memory and executable on the processor. When the vision and IMU-based positioning program is executed by the processor, the steps of the vision and IMU-based positioning method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a positioning program based on vision and IMU, and when the positioning program based on vision and IMU is executed by a processor, the steps of the positioning method based on vision and IMU as described in any one of claims 1 to 7 are implemented.

Images