Training method, pose estimation method and device for visual inertial odometry model
Through a deep learning framework, the visual inertial odometer model is trained, and the visual and inertial information is tightly coupled, which solves the problems of initialization complexity and tracking loss in the prior art, and improves the accuracy and robustness of pose estimation.
Patent Information
- Application Number
- CN202210195781.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-01
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-03-01
AI Technical Summary
The existing visual inertial odometry methods are complex and time-consuming in the initialization and optimization process, prone to tracking loss, and deep learning methods cannot recover the motion scale and the visual and inertial information are not fully integrated, resulting in low accuracy of pose estimation.
The visual inertial odometer model is trained using a deep learning framework. By obtaining sample image sets and IMU data sets, the visual and inertial information are tightly coupled through the attention mechanism. Weighted fusion feature encoding to determine the target loss function to adjust network parameters.
The initialization and optimization process is simplified, tracking loss is avoided, the accuracy and robustness of pose estimation is improved, the full fusion of visual and inertial information is achieved, and the performance of the visual inertial odometer model is improved.
Smart Images

Figure CN114612556B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a training method, a pose estimation method and a device for a visual inertial odometer model. Background Art
[0002] Visual odometry is a submodule in the visual SLAM (Simultaneous localization and mapping) problem. It uses two adjacent frames of images taken by the camera on the robot during its movement to calculate the relative pose between the two frames. Visual odometry only uses the camera as a sensor and is greatly affected by optical properties, such as lighting changes, moving objects, and textureless areas. Some existing methods use inertial sensors IMU (Inertial Measure Unit) as a supplement to vision, design visual inertial odometry, use inertial sensors to measure the acceleration and angular velocity of the robot, and integrate visual information to obtain a more robust estimation effect.
[0003] Current visual inertial odometry mostly uses nonlinear optimization to fuse camera and inertial sensor information in a tightly coupled form. However, this approach has complex initialization and calibration, a long or divergent optimization iteration process, and may cause tracking loss (i.e., the pose cannot be estimated). Existing visual inertial odometry using deep learning cannot restore the motion scale like traditional methods, and on the other hand, it treats the visual and inertial parts as independent modules. They do not fully fuse visual and inertial information, and the final pose estimation accuracy is worse than that of traditional nonlinear optimization algorithms. Summary of the invention
[0004] The main technical problem solved by this application is to provide a training method, a pose estimation method and related devices for a visual inertial odometer model.
[0005] In a first aspect, the present application provides a training method for a visual inertial odometry model, which includes: obtaining a sample image set and a sample IMU data set; wherein the sample image set includes a plurality of frames of continuous sample color images obtained by an image acquisition device, and the sample IMU data set includes corresponding sample IMU data obtained when the plurality of frames of continuous sample color images are obtained; inputting two adjacent frames of sample color images in the sample image set and the sample IMU data corresponding to the two adjacent frames of sample color images into the visual inertial odometry model, and outputting two frames of depth images corresponding to the two adjacent frames of sample color images and an estimated pose when the image acquisition device acquires the two adjacent frames of sample color images; determining a target loss function of the visual inertial odometry model based on the two frames of depth images corresponding to the two adjacent frames of sample color images, the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images, and the sample IMU data corresponding to the two adjacent frames of sample color images; and adjusting the network parameters of the visual inertial odometry model using the target loss function.
[0006] Therefore, by acquiring a sample image set and a sample IMU data set, wherein the sample image set includes several frames of continuous sample color images acquired by an image acquisition device, and the sample IMU data set includes corresponding sample IMU data acquired when acquiring several frames of continuous sample color images, after the sample image set and the sample IMU data set are input into the visual inertial odometry model, the visual inertial odometry model can be used to estimate the scene depth and the pose of the image acquisition device. Specifically, two frames of depth images corresponding to two adjacent frames of sample color images and the estimated pose when the image acquisition device acquires two adjacent frames of sample color images can be output. Therefore, based on the two frames of depth images corresponding to the two adjacent frames of sample color images and the image acquisition device acquiring the adjacent frames of sample color images, the scene depth and the pose of the image acquisition device can be estimated. The target loss function of the visual inertial odometry model is determined by the estimated pose between two frames of sample color images and the sample IMU data corresponding to two adjacent frames of sample color images. Therefore, the visual information and IMU information are integrated in the network. By utilizing their respective advantages, a more accurate and robust visual inertial odometry model can be obtained. In addition, the visual inertial odometry is implemented using a deep learning framework. Compared with the traditional nonlinear method based on BA (Bundle-Adjustment), there is no need for complex initialization and iteration process, and the model is simpler. This solves the complex initialization and optimization problems in the traditional nonlinear optimization algorithm based on BA, and avoids tracking loss in complex scenes.
[0007] The visual inertial odometry model comprises a depth estimation network, a visual encoding network, an IMU encoding network and a visual inertial fusion network; the step of inputting two adjacent frames of sample color images in the sample image set and the sample IMU data corresponding to the two adjacent frames of sample color images into the visual inertial odometry model, and outputting two frames of depth images corresponding to the two adjacent frames of sample color images and the estimated position and posture when the image acquisition device acquires the two adjacent frames of sample color images comprises: inputting the sample color images in the sample image set into the depth estimation network to obtain the depth images corresponding to the sample color images; and superimposing the previous frame of sample color images and the current frame of sample color images in the sample image set and inputting them into the visual encoding network to obtain visual feature coding; inputting the sample IMU data corresponding to the previous frame of sample color images and the current frame of sample color images into the IMU encoding network to obtain IMU feature coding; and inputting the visual feature coding and the IMU feature coding into the visual inertial fusion network to obtain the estimated position and posture when the image acquisition device acquires the current frame of sample color images.
[0008] Therefore, a visual inertial odometry model is composed of a depth estimation network, a visual coding network, an IMU coding network and a visual-inertial fusion network. The sample color image in the sample image set is input into the depth estimation network to obtain the depth image corresponding to the sample color image, thereby realizing the estimation of the depth map of the environment in which the image acquisition device is located; the visual feature coding is obtained by superimposing the previous frame sample color image and the current frame sample color image in the sample image set and inputting them into the visual coding network, and at the same time, the sample IMU data corresponding to the previous frame sample color image and the current frame sample color image are input into the IMU coding network to obtain the IMU feature coding, and then the visual feature coding and the IMU feature coding are input into the visual-inertial fusion network, so as to obtain the estimated posture of the image acquisition device when obtaining the current frame sample color image, thereby realizing the estimation of the posture of the image acquisition device itself.
[0009] In which, the depth estimation network includes an encoder and a decoder connected to each other; the step of inputting a sample color image in the sample image set into the depth estimation network to obtain a depth image corresponding to the sample color image includes: inputting the sample color image into the depth estimation network, using the downsampling layer of the encoder to transform the sample color image into a depth feature map, and then using the upsampling layer of the decoder to transform the depth feature map into a depth image corresponding to the sample color image.
[0010] Therefore, by inputting the sample color image into the depth estimation network, the depth estimation network adopts an encoder-decoder structure, uses the downsampling layer of the encoder to transform the sample color image into a depth feature map, and then uses the upsampling layer of the decoder to transform the depth feature map into a depth image corresponding to the sample color image, so that the deep learning framework can be used to realize the estimation of the depth map of the environment in which the image acquisition device is located.
[0011] Among them, the visual-inertial fusion network adopts an attention mechanism, and the visual-inertial fusion network includes a feedforward neural network; the visual feature code and the IMU feature code are input into the visual-inertial fusion network to obtain the estimated pose of the image acquisition device when acquiring the current frame sample color image, including: weighted fusion of the visual feature code and the IMU feature code through the attention mechanism to obtain an optimized feature code; and the optimized feature code is processed by using a feedforward neural network to obtain the estimated pose of the image acquisition device when acquiring the current frame sample color image.
[0012] Therefore, the visual feature coding and the IMU feature coding are weightedly fused through the attention mechanism to obtain the optimized feature coding, and the optimized feature coding is processed by the feedforward neural network to obtain the estimated pose of the image acquisition device when acquiring the current frame sample color image. Since the attention mechanism focuses on the complementarity of visual information and IMU information, that is, IMU information can provide better motion estimation for short-term rapid motion, and visual information will not drift compared to IMU information, so in different scenarios, the attention mechanism can effectively learn the relationship between visual features and inertial features, making the performance of the visual inertial odometry model more robust in different scenarios.
[0013] Among them, the visual-inertial fusion network also includes a first multi-layer perceptron and a second multi-layer perceptron; the visual feature code and the IMU feature code are weightedly fused through the attention mechanism to obtain the optimized feature code, including: inputting the IMU feature code into the first multi-layer perceptron and the second multi-layer perceptron respectively to obtain a plurality of key-value pairs, each of which includes a key and a value; obtaining the similarity between the visual feature code and the key in each key-value pair, using the similarity as a weight, multiplying the weight by the value in the corresponding key-value pair and summing them to obtain the optimized feature code.
[0014] Therefore, by inputting the IMU feature code into the first multi-layer perceptron and the second multi-layer perceptron respectively, several key-value pairs can be obtained, each of which includes a key and a value, and then the similarity between the visual feature code and the key in each key-value pair is obtained, and the similarity is used as a weight to multiply the value in the corresponding key-value pair and then summed to obtain the optimized feature code, and then the optimized feature code can be used to obtain the estimated pose when the image acquisition device obtains the current frame sample color image. Therefore, by using a tightly coupled visual-inertial fusion network to fuse the visual feature code and the IMU feature code, compared with the existing loosely coupled network that directly splices the two codes, the accuracy of the visual information and the high frequency of the IMU information can be fully utilized, so that the visual information and the IMU information can be better complementary.
[0015] Among them, the target loss function includes a depth loss function, a photometric loss function and an IMU loss function; the target loss function of the visual inertial odometer model is determined based on the two depth images corresponding to the two adjacent frames of sample color images, the estimated pose when the image acquisition device obtains the two adjacent frames of sample color images, and the sample IMU data corresponding to the two adjacent frames of sample color images, including: determining the depth loss function according to the depth image corresponding to the previous frame of sample color image and the depth image corresponding to the current frame of sample color image; and determining the photometric loss function according to the estimated pose when the image acquisition device obtains the current frame of sample color image and the depth image corresponding to the current frame of sample color image; and determining the IMU loss function according to the estimated pose when the image acquisition device obtains the current frame of sample color image and the sample IMU data corresponding to the previous frame of sample color image and the current frame of sample color image.
[0016] Therefore, in the training process of the visual inertial odometry model, the target loss functions of the visual inertial odometry model used include depth loss function, photometric loss function and IMU loss function. The photometric difference and depth map difference of the transformed two frames of images are calculated according to the estimated pose and depth, and the visual photometric error and geometric error are calculated. The depth estimation and pose estimation are constrained by using the visual depth loss function and photometric loss function. At the same time, the IMU error is calculated according to the difference between the result calculated by the IMU's own kinematics and the pose estimation result. The two constraints of the IMU itself, namely the speed constraint and the position constraint, are used to associate the pose predicted by the network with the physical properties of the IMU, which can make the visual inertial odometry model training process converge faster and obtain the absolute scale.
[0017] In order to solve the above problems, the second aspect of the present application provides a pose estimation method, which comprises: using an image acquisition device to acquire several frames of continuous target color images, and determining the target IMU data corresponding to the time when the image acquisition device acquires the several frames of continuous target color images; inputting the several frames of continuous target color images and the corresponding target IMU data into a visual inertial odometry model to obtain an estimated pose when the image acquisition device acquires the target color images; wherein the visual inertial odometry model is trained using the training method of the visual inertial odometry model of the first aspect mentioned above.
[0018] Therefore, an image acquisition device is used to acquire several frames of continuous target color images, and the target IMU data corresponding to the image acquisition device when acquiring the several frames of continuous target color images is determined. By inputting the several frames of continuous target color images and the corresponding target IMU data into the visual inertial odometry model, the estimated pose when the image acquisition device acquires the target color images is obtained. Since the visual inertial odometry model is trained using the training method of the visual inertial odometry model of the first aspect above, that is, the visual inertial odometry model integrates visual information and IMU information in the network, a more accurate and robust pose estimation result can be obtained by utilizing the respective advantages of the two.
[0019] In order to solve the above problems, the third aspect of the present application provides a training device for a visual inertial odometry model, the training device for the visual inertial odometry model comprising: a sample acquisition module, used to acquire a sample image set and a sample IMU data set; wherein the sample image set comprises a plurality of frames of continuous sample color images acquired by an image acquisition device, and the sample IMU data set comprises corresponding sample IMU data acquired when acquiring the plurality of frames of continuous sample color images; a processing module, used to input the sample IMU data corresponding to two adjacent frames of sample color images in the sample image set and the sample IMU data corresponding to the two adjacent frames of sample color images into a processing module; The visual inertial odometry model outputs two frames of depth images corresponding to the two adjacent frames of sample color images and the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images; a loss function determination module is used to determine the target loss function of the visual inertial odometry model based on the two frames of depth images corresponding to the two adjacent frames of sample color images, the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images, and the sample IMU data corresponding to the two adjacent frames of sample color images; a parameter adjustment module is used to adjust the network parameters of the visual inertial odometry model using the target loss function.
[0020] In order to solve the above problems, the fourth aspect of the present application provides a pose estimation device, which includes: a data acquisition module, which is used to use an image acquisition device to acquire several frames of continuous target color images, and determine the target IMU data corresponding to the image acquisition device when acquiring the several frames of continuous target color images; a pose estimation module, which is used to input the several frames of continuous target color images and the corresponding target IMU data into a visual inertial odometry model to obtain an estimated pose when the image acquisition device acquires the target color images; wherein the visual inertial odometry model is trained using the training method of the visual inertial odometry model of the first aspect above.
[0021] In order to solve the above problems, the fifth aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the training method of the visual inertial odometry model in the above first aspect, or the pose estimation method in the above second aspect.
[0022] In order to solve the above problems, the sixth aspect of the present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the training method of the visual inertial odometry model in the first aspect or the pose estimation method in the second aspect is implemented.
[0023] The above scheme obtains a sample image set and a sample IMU data set, wherein the sample image set includes several frames of continuous sample color images obtained by using an image acquisition device, and the sample IMU data set includes corresponding sample IMU data obtained when obtaining several frames of continuous sample color images. After the sample image set and the sample IMU data set are input into the visual inertial odometry model, the visual inertial odometry model can be used to estimate the scene depth and the pose of the image acquisition device. Specifically, two frames of depth images corresponding to two adjacent frames of sample color images and the estimated pose when the image acquisition device obtains two adjacent frames of sample color images can be output. Therefore, based on the two frames of depth images corresponding to the two adjacent frames of sample color images and the image acquisition device obtaining the corresponding The target loss function of the visual inertial odometry model is determined by the estimated pose between two adjacent frames of sample color images and the sample IMU data corresponding to the two adjacent frames of sample color images. Therefore, the visual information and IMU information are integrated in the network. By utilizing the advantages of both, a more accurate and robust visual inertial odometry model can be obtained. In addition, the visual inertial odometry is implemented using a deep learning framework. Compared with the traditional nonlinear method based on BA (Bundle-Adjustment), there is no need for complex initialization and iteration process, and the model is simpler. This solves the complex initialization and optimization problems in the traditional nonlinear optimization algorithm based on BA, and avoids tracking loss in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a flowchart of an embodiment of a training method for a visual inertial odometry model of the present application;
[0025] Figure 2 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S12;
[0026] Figure 3 It is a schematic diagram of the principle of obtaining a depth image through a depth estimation network in an application scenario;
[0027] Figure 4 yes Figure 2 A schematic diagram of a flow chart of an embodiment of step S124;
[0028] Figure 5 This is a schematic diagram of the principle of obtaining estimated pose through a visual-inertial fusion network in an application scenario;
[0029] Figure 6 yes Figure 4 A flow chart of an embodiment of step S1241;
[0030] Figure 7 This is a schematic diagram of the principle of obtaining key-value pairs through IMU feature encoding in an application scenario;
[0031] Figure 8 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S13;
[0032] Fig. 9 This is a schematic diagram of the principle of the training process of the visual inertial odometry model in an application scenario;
[0033] Fig.10 It is a flowchart of an embodiment of a posture estimation method of the present application;
[0034] Fig.11 This is a schematic diagram of the principle of achieving pose estimation through a visual inertial odometry model in an application scenario;
[0035] Fig.12 It is a schematic diagram of the framework of an embodiment of a training device for a visual inertial odometry model of the present application;
[0036] Fig.13 It is a schematic diagram of the framework of an embodiment of a posture estimation device of the present application;
[0037] Fig.14 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;
[0038] Fig.15 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION
[0039] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.
[0040] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.
[0041] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.
[0042] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a training method for a visual inertial odometry model of the present application. Specifically, the following steps may be included:
[0043] Step S11: Acquire a sample image set and a sample IMU data set, wherein the sample image set includes a plurality of frames of continuous sample color images acquired by an image acquisition device, and the sample IMU data set includes corresponding sample IMU data acquired when acquiring the plurality of frames of continuous sample color images.
[0044] Specifically, an image acquisition device can be used to acquire a sample image set and a sample IMU data set. For example, during the movement of the image acquisition device, several frames of continuous sample color images can be acquired as a sample image set, and an inertial navigation device is provided on the image acquisition device or on the motion device where the image acquisition device is located. During the process of the image acquisition device acquiring several frames of continuous sample color images, the inertial navigation device can synchronously acquire corresponding sample IMU data as a sample IMU data set.
[0045] Step S12: input two adjacent frames of sample color images in the sample image set and the sample IMU data corresponding to the two adjacent frames of sample color images into the visual inertial odometer model, output two frames of depth images corresponding to the two adjacent frames of sample color images and the estimated posture when the image acquisition device obtains the two adjacent frames of sample color images.
[0046] Specifically, two adjacent frames of sample color images are selected from several frames of continuous sample color images in the sample image set, for example, the current frame sample color image and the previous frame sample color image are selected, and then the sample IMU data between the current frame sample color image and the previous frame sample color image are found from the sample IMU data set. Since the sampling frequency of IMU data is generally greater than the sampling frequency of the image, there may be multiple groups of sample IMU data corresponding to the two adjacent frames of sample color images. Therefore, the selected two adjacent frames of sample color images and all the sample IMU data corresponding to the two adjacent frames of sample color images constitute the current batch of training data and are input into the visual inertial odometer model, and the two frames of depth images corresponding to the two adjacent frames of sample color images and the estimated pose when the image acquisition device obtains the two adjacent frames of sample color images are output.
[0047] Step S13: Determine the target loss function of the visual inertial odometer model based on the two frames of depth images corresponding to the two adjacent frames of sample color images, the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images, and the sample IMU data corresponding to the two adjacent frames of sample color images.
[0048] Step S14: using the target loss function to adjust the network parameters of the visual inertial odometry model.
[0049] It can be understood that after obtaining the two depth images corresponding to two adjacent frames of sample color images, the estimated pose when the image acquisition device obtains the two adjacent frames of sample color images, and all the sample IMU data corresponding to the two adjacent frames of sample color images, the photometric difference and depth map difference after the transformation of the two frames of images can be calculated respectively according to the estimated pose and depth to obtain the visual photometric error and geometric error. The IMU error can also be calculated based on the difference between the results of the IMU's own kinematic calculation and the pose estimation results, so that the target loss function of the visual inertial odometry model can be determined.
[0050] It is understandable that after obtaining the target loss function, the network parameters of the visual inertial odometry model can be adjusted according to the target loss function to update the visual inertial odometry model. In addition, during the training process of the visual inertial odometry model, the convergence of the target loss function can be obtained. When the target loss function converges, the update of the network parameters of the visual inertial odometry model can be stopped. When the target loss function does not converge, the number of adjustments of the network parameters can be obtained. When the number of adjustments reaches a preset number, the final visual inertial odometry model can be determined according to the network parameters at this time to avoid the loss function not converging and affecting the training efficiency.
[0051] The above scheme obtains a sample image set and a sample IMU data set, wherein the sample image set includes a plurality of frames of continuous sample color images obtained by using an image acquisition device, and the sample IMU data set includes corresponding sample IMU data obtained when obtaining a plurality of frames of continuous sample color images. After the sample image set and the sample IMU data set are input into the visual inertial odometer model, the visual inertial odometer model can be used to estimate the scene depth and the pose of the image acquisition device. Specifically, two frames of depth images corresponding to two adjacent frames of sample color images and the estimated pose when the image acquisition device obtains two adjacent frames of sample color images can be output. Therefore, based on the two frames of depth images corresponding to the two adjacent frames of sample color images and the image acquisition device obtaining the adjacent frames of sample color images, the scene depth and the pose of the image acquisition device can be estimated. The target loss function of the visual inertial odometry model is determined by the estimated pose between two frames of sample color images and all sample IMU data corresponding to two adjacent frames of sample color images. Therefore, the visual information and IMU information are integrated in the network. By utilizing the advantages of both, a more accurate and robust visual inertial odometry model can be obtained. In addition, the visual inertial odometry is implemented using a deep learning framework. Compared with the traditional nonlinear method based on BA (Bundle-Adjustment), there is no need for complex initialization and iteration process, and the model is simpler. This solves the complex initialization and optimization problems in the traditional nonlinear optimization algorithm based on BA, and avoids tracking loss in complex scenes.
[0052] See also Figure 2 , Figure 2 yes Figure 1 Flow chart of an embodiment of step S12 in FIG. In this embodiment, the visual inertial odometer model includes a depth estimation network, a visual encoding network, an IMU encoding network and a visual inertial fusion network; the above step S12 may specifically include the following steps:
[0053] Step S121: inputting a sample color image in the sample image set into the depth estimation network to obtain a depth image corresponding to the sample color image.
[0054] Specifically, the sample color image of the previous frame is input into the depth estimation network to obtain the depth image corresponding to the sample color image of the previous frame, and the sample color image of the current frame is input into the depth estimation network to obtain the depth image corresponding to the sample color image of the current frame. Taking the sample color image of the current frame as an example, the input of the depth estimation network is the sample color image (RGB image) of the current frame, with a size of H*W*3, and the output of the depth estimation network is the depth image corresponding to the sample color image of the current frame predicted by the network, with a size of H*W*1.
[0055] In one embodiment, the depth estimation network includes an encoder and a decoder connected to each other; the above step S121 may specifically include: inputting the sample color image into the depth estimation network, using the downsampling layer of the encoder to transform the sample color image into a depth feature map, and then using the upsampling layer of the decoder to transform the depth feature map into a depth image corresponding to the sample color image.
[0056] Please combine Figure 3 , Figure 3 This is a schematic diagram of the principle of obtaining a depth image through a depth estimation network in an application scenario. The depth estimation network adopts an encoder-decoder structure. The sample color image is input into the depth estimation network. In the encoder (Encoder), the sample color image is transformed into a feature map of H / 64*W / 64*1024 through a downsampling layer, and in the decoder (Decoder), the feature map is used to obtain a depth image of the same size as the sample color image through an upsampling layer. Therefore, the deep learning framework can be used to realize the estimation of the dense depth map of the environment in which the image acquisition device is located.
[0057] Step S122: superimpose the previous frame sample color image and the current frame sample color image in the sample image set and input them into the visual coding network to obtain visual feature coding.
[0058] Step S123: inputting the sample IMU data corresponding to the previous frame sample color image and the current frame sample color image into the IMU encoding network to obtain IMU feature encoding.
[0059] Step S124: inputting the visual feature code and the IMU feature code into the visual-inertial fusion network to obtain an estimated pose when the image acquisition device acquires the current frame sample color image.
[0060] It can be understood that the visual coding network can use the image information of the sample color image to obtain the code containing the pixel motion and camera motion information, and the IMU coding network can use the IMU data between the previous frame sample color image and the current frame sample color image to obtain the code with the same number of channels as the output of the visual coding network. For example, the input of the visual coding network is the superposition of the current frame sample color image and the previous frame sample color image, with a size of H*W*6, and the output is the visual feature code with a size of 1*1024; the input of the IMU coding network is all the IMU data between the current frame sample color image and the previous frame sample color image, and the IMU data may include acceleration data and angular velocity data. When the frequency of the IMU is 10 times that of the image acquisition device, the input size of the IMU coding network is 10*6 and the output size is 10*1024. The visual inertial fusion network adopts a method similar to the tight coupling in traditional optimization, using the visual feature code output by the visual coding network and the IMU feature code output by the IMU coding network to obtain the fused code, and finally outputs the estimated pose.
[0061] The above scheme utilizes a depth estimation network, a visual coding network, an IMU coding network and a visual-inertial fusion network to form a visual-inertial odometer model. The sample color image in the sample image set is input into the depth estimation network to obtain the depth image corresponding to the sample color image, thereby realizing the estimation of the depth map of the environment in which the image acquisition device is located; the visual feature coding is obtained by superimposing the previous frame sample color image and the current frame sample color image in the sample image set and inputting them into the visual coding network. At the same time, all the sample IMU data corresponding to the previous frame sample color image and the current frame sample color image are input into the IMU coding network to obtain the IMU feature coding. After that, the visual feature coding and the IMU feature coding are input into the visual-inertial fusion network, and the estimated posture of the image acquisition device when acquiring the current frame sample color image can be obtained, thereby realizing the estimation of the posture of the image acquisition device itself.
[0062] See also Figure 4 , Figure 4 yes Figure 2 Schematic diagram of a flow chart of an embodiment of step S124 in FIG. In this embodiment, the visual-inertial fusion network adopts an attention mechanism, and the visual-inertial fusion network includes a feedforward neural network; the above step S124 may specifically include the following steps:
[0063] Step S1241: weighted fusion of the visual feature code and the IMU feature code is performed through an attention mechanism to obtain an optimized feature code.
[0064] Step S1242: Process the optimized feature encoding using a feedforward neural network to obtain an estimated pose when the image acquisition device acquires the current frame sample color image.
[0065] Please combine Figure 5 , Figure 5 This is a schematic diagram of the principle of obtaining estimated pose through a visual-inertial fusion network in an application scenario. The input accepted by the visual-inertial fusion network is the output of the visual encoding network and the IMU encoding network, with sizes of 1*1024 and 10*1024 respectively. The visual-inertial fusion network finally outputs a pose estimate with a size of 1*6, where 6 represents the prediction of the relative pose with 6 degrees of freedom, including a 3-dimensional translation vector and a 3-dimensional Euler angle. The visual-inertial fusion network adopts the attention mechanism to fuse the visual features and inertial features. As an important component of the neural network structure, the attention mechanism can suppress useless features in the channel and enhance the features that need to be used, so that the overall network can better perform feature selection and fusion; "Key, Value" is called a key-value pair, where Key is the key and Value is the value, which is obtained by processing the IMU feature code. The k*1024 (for example, k=10, which is related to the frequency of the IMU and the image) IMU feature codes are split into 10 1*1024 codes, and the similarity between the visual feature code (Visual Code) and each key (Key) is calculated. The similarity is multiplied by each value (Value) as a weight, and then the results are added to obtain an optimized code (RefinedCode) of size 1*1024, that is, the optimized feature code. The optimized feature code is passed through the feedforward neural network (Feed Forword) to finally obtain a 1*6 pose Pose.
[0066] Furthermore, please combine Figure 6 , Figure 6 yes Figure 4 In the flowchart of step S1241 of an embodiment, in one embodiment, the visual-inertial fusion network further includes a first multi-layer perceptron and a second multi-layer perceptron; the above step S1241 may specifically include:
[0067] Step S12411: input the IMU feature code into the first multilayer perceptron and the second multilayer perceptron respectively to obtain a plurality of key-value pairs, each of which includes a key and a value.
[0068] Step S12412: Obtain the similarity between the visual feature code and the key in each key-value pair, use the similarity as a weight, multiply the weight by the value in the corresponding key-value pair and then sum them to obtain the optimized feature code.
[0069] Please combine Figure 7 , Figure 7 It is a schematic diagram of the principle of obtaining key-value pairs through IMU feature coding in an application scenario. The key-value pairs are formed by IMU feature coding through different multi-layer perceptrons or fully connected layers. By inputting the IMU feature coding into the first multi-layer perceptron and the second multi-layer perceptron respectively, several key-value pairs can be obtained, each of which includes a key and a value, and then the similarity between the visual feature coding and the key in each key-value pair is obtained, and the similarity is multiplied by the value in the corresponding key-value pair as a weight and then summed to obtain the optimized feature coding, and then the optimized feature coding can be used to obtain the estimated posture of the image acquisition device when acquiring the current frame sample color image. Therefore, by using a tightly coupled visual-inertial fusion network to fuse the visual feature coding and the IMU feature coding, compared with the existing loosely coupled network that directly splices the two codes, the accuracy of the visual information and the high frequency of the IMU information can be fully utilized, so that the visual information and the IMU information can be better complementary.
[0070] The process of weighted fusion of visual feature encoding and key-value pairs to obtain optimized feature encoding can be expressed by the following formula:
[0071]
[0072] Among them, Q represents the visual feature encoding, d is the normalization parameter, and K i is the ith key, V i Different from the traditional method of directly weighting and averaging the visual feature encoding and the IMU feature encoding, the fusion method in the embodiment of the present application fully utilizes the accuracy of vision and the high frequency of IMU, and is a tightly coupled fusion method.
[0073] The above scheme uses the attention mechanism to perform weighted fusion of visual feature coding and IMU feature coding to obtain optimized feature coding, and uses a feedforward neural network to process the optimized feature coding, so as to obtain the estimated pose when the image acquisition device obtains the current frame sample color image. Since the attention mechanism focuses on the complementarity of visual information and IMU information, that is, IMU information can provide better motion estimation for short-term rapid motion, and visual information will not drift compared to IMU information, so in different scenarios, the attention mechanism can effectively learn the relationship between visual features and inertial features, so that the performance of the visual inertial odometer model in different scenarios is more robust.
[0074] See also Figure 8 , Figure 8 yes Figure 1 Schematic diagram of the flow chart of an embodiment of step S13 in FIG. In this embodiment, the target loss function includes a depth loss function, a photometric loss function and an IMU loss function; the above step S13 may specifically include the following steps:
[0075] Step S131: determining the depth loss function according to the depth image corresponding to the previous frame sample color image and the depth image corresponding to the current frame sample color image.
[0076] Step S132: Determine the photometric loss function according to the estimated pose when the image acquisition device acquires the current frame sample color image and the depth image corresponding to the current frame sample color image.
[0077] Step S133: Determine the IMU loss function according to the estimated posture when the image acquisition device acquires the current frame sample color image and the sample IMU data corresponding to the previous frame sample color image and the current frame sample color image.
[0078] Please combine Fig. 9 , Fig. 9 This is a schematic diagram of the principle of the training process of the visual inertial odometry model in an application scenario. The visual inertial odometry model includes a depth estimation network (DepthNet), a visual encoding network (VisualOdom), an IMU encoding network (IMUOdom) and a visual inertial fusion network (Sensor Fusion). t-1 ) corresponds to the depth image and the current frame sample color image (I t ) corresponds to the depth image Determine the depth loss function (Loss geo ), according to the estimated pose (R, t) when the image acquisition device obtains the current frame sample color image and the depth image corresponding to the current frame sample color image Determine the photometric loss function (Loss pho ), according to the estimated pose (R, t) when the image acquisition device obtains the current frame sample color image and all the sample IMU data (IMU t-1,t ), determine the IMU loss function (Loss imu ), where the IMU loss function may include a velocity loss function and a displacement loss function.
[0079] During the training process of the visual inertial odometry model, the target loss functions of the visual inertial odometry model used include depth loss function, photometric loss function and IMU loss function. The photometric difference and depth map difference of the transformed two frames of images are calculated according to the estimated pose and depth, and the visual photometric error and geometric error are calculated. The depth estimation and pose estimation are constrained by using the visual depth loss function and photometric loss function. At the same time, the IMU error is calculated according to the difference between the result calculated by the IMU's own kinematics and the pose estimation result. The two constraints of the IMU itself, namely the speed constraint and the position constraint, are used to associate the pose predicted by the network with the physical properties of the IMU, which can make the training process of the visual inertial odometry model converge faster and obtain the absolute scale.
[0080] See also Fig.10 , Fig.10 This is a flow chart of an embodiment of the pose estimation method of the present application. Specifically, it may include the following steps:
[0081] Step S101: using an image acquisition device to acquire a plurality of frames of continuous target color images, and determining target IMU data corresponding to when the image acquisition device acquires the plurality of frames of continuous target color images.
[0082] Step S102: inputting the plurality of frames of continuous target color images and corresponding target IMU data into a visual inertial odometer model to obtain an estimated position and posture when the image acquisition device acquires the target color image.
[0083] The visual inertial odometry model is trained using any of the above-mentioned training methods for the visual inertial odometry model.
[0084] Please combine Fig.11 , Fig.11It is a schematic diagram of the principle of realizing pose estimation through a visual inertial odometry model in an application scenario. The collected target color image and the corresponding target IMU data are input into the visual inertial odometry model. The visual inertial odometry model includes a depth estimation network, a visual encoding network, an IMU encoding network and a visual inertial fusion network, and can output a dense depth map D and a camera pose (R, t). It can be understood that, an image acquisition device is used to acquire several frames of continuous target color images, and the corresponding target IMU data when the image acquisition device acquires several frames of continuous target color images is determined. By inputting several frames of continuous target color images and the corresponding target IMU data into the visual inertial odometry model, the estimated pose when the image acquisition device acquires the target color image is obtained. Since the visual inertial odometry model is trained using the training method of the visual inertial odometry model of the first aspect above, that is, the visual inertial odometry model fuses visual information with IMU information in the network, a more accurate and robust pose estimation result can be obtained by utilizing the respective advantages of the two.
[0085] Furthermore, the visual inertial odometry model can be used to obtain an estimated pose when the image acquisition device acquires continuous target color images, and the motion trajectory estimation of the image acquisition device can be obtained by connecting the continuous estimated poses.
[0086] The execution subject of the pose estimation method of the present application may be a pose estimation device, for example, the pose estimation method may be executed by a positioning device or a server or other processing device, wherein the positioning device may be a mobile device such as a robot, an unmanned vehicle, or an unmanned aerial vehicle, or may be a user equipment (UE), a user terminal, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the pose estimation method may be implemented by a processor calling computer-readable instructions stored in a memory.
[0087] See also Fig.12 , Fig.12It is a framework diagram of an embodiment of a training device for a visual inertial odometry model of the present application. In this embodiment, the training device 120 for a visual inertial odometry model includes: a sample acquisition module 1200, used to acquire a sample image set and a sample IMU data set; wherein the sample image set includes a plurality of frames of continuous sample color images acquired by an image acquisition device, and the sample IMU data set includes corresponding sample IMU data acquired when acquiring the plurality of frames of continuous sample color images; a processing module 1202, used to input two adjacent frames of sample color images in the sample image set and the sample IMU data corresponding to the two adjacent frames of sample color images into the visual inertial odometry model, and output The two frames of depth images corresponding to the two adjacent frames of sample color images and the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images; a loss function determination module 1204, used to determine the target loss function of the visual inertial odometry model based on the two frames of depth images corresponding to the two adjacent frames of sample color images, the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images and the sample IMU data corresponding to the two adjacent frames of sample color images; a parameter adjustment module 1206, used to adjust the network parameters of the visual inertial odometry model using the target loss function.
[0088] In the above scheme, a sample image set and a sample IMU data set are acquired through a sample acquisition module 1200, wherein the sample image set includes a plurality of frames of continuous sample color images acquired by an image acquisition device, and the sample IMU data set includes corresponding sample IMU data acquired when acquiring a plurality of frames of continuous sample color images. After the processing module 1202 inputs the sample image set and the sample IMU data set into the visual inertial odometer model, the visual inertial odometer model can be used to estimate the scene depth and the pose of the image acquisition device. Specifically, two frames of depth images corresponding to two adjacent frames of sample color images and the estimated pose when the image acquisition device acquires two adjacent frames of sample color images can be output. Therefore, the loss function determination module 1204 can be based on the two frames corresponding to the two adjacent frames of sample color images. The depth image and image acquisition device obtain the estimated pose when two adjacent frames of sample color images and all sample IMU data corresponding to the two adjacent frames of sample color images to determine the target loss function of the visual inertial odometry model. Therefore, the visual information and IMU information are integrated in the network, and the advantages of both are utilized to obtain a more accurate and robust visual inertial odometry model. In addition, the visual inertial odometry is implemented using a deep learning framework. Compared with the traditional nonlinear method based on BA (Bundle-Adjustment), there is no need for complex initialization and iteration process, and the model is simpler, which solves the complex initialization and optimization problems in the traditional nonlinear optimization algorithm based on BA, and avoids tracking loss in complex scenes.
[0089] In some embodiments, the visual inertial odometry model includes a depth estimation network, a visual encoding network, an IMU encoding network and a visual inertial fusion network; the processing module 1202 executes the step of inputting two adjacent frames of sample color images in the sample image set and the sample IMU data corresponding to the two adjacent frames of sample color images into the visual inertial odometry model, and outputting two frames of depth images corresponding to the two adjacent frames of sample color images and the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images, specifically including: inputting the sample color images in the sample image set into the depth estimation network to obtain the depth images corresponding to the sample color images; and superimposing the previous frame of sample color images and the current frame of sample color images in the sample image set and inputting them into the visual encoding network to obtain visual feature coding; inputting the sample IMU data corresponding to the previous frame of sample color images and the current frame of sample color images into the IMU encoding network to obtain IMU feature coding; inputting the visual feature coding and the IMU feature coding into the visual inertial fusion network to obtain the estimated pose when the image acquisition device acquires the current frame of sample color images.
[0090] In some embodiments, the depth estimation network includes an encoder and a decoder connected to each other; the processing module 1202 executes the step of inputting the sample color image in the sample image set into the depth estimation network to obtain a depth image corresponding to the sample color image, specifically including: inputting the sample color image into the depth estimation network, using the downsampling layer of the encoder to transform the sample color image into a depth feature map, and then using the upsampling layer of the decoder to transform the depth feature map into a depth image corresponding to the sample color image.
[0091] In some embodiments, the visual-inertial fusion network adopts an attention mechanism, and the visual-inertial fusion network includes a feedforward neural network; the processing module 1202 executes the step of inputting the visual feature code and the IMU feature code into the visual-inertial fusion network to obtain the estimated pose of the image acquisition device when acquiring the current frame sample color image, specifically including: weightedly fusing the visual feature code and the IMU feature code through the attention mechanism to obtain an optimized feature code; and processing the optimized feature code using a feedforward neural network to obtain the estimated pose of the image acquisition device when acquiring the current frame sample color image.
[0092] In some embodiments, the visual-inertial fusion network also includes a first multi-layer perceptron and a second multi-layer perceptron; the processing module 1202 performs a step of weighted fusion of the visual feature code and the IMU feature code through an attention mechanism to obtain an optimized feature code, which may specifically include: inputting the IMU feature code into the first multi-layer perceptron and the second multi-layer perceptron respectively to obtain a plurality of key-value pairs, each of which includes a key and a value; obtaining the similarity between the visual feature code and the key in each key-value pair, using the similarity as a weight, multiplying the weight by the value in the corresponding key-value pair and summing them to obtain the optimized feature code.
[0093] In some embodiments, the target loss function includes a depth loss function, a photometric loss function and an IMU loss function; the loss function determination module 1204 executes the step of determining the target loss function of the visual inertial odometer model based on the two depth images corresponding to the two adjacent frames of sample color images, the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images, and the sample IMU data corresponding to the two adjacent frames of sample color images, including: determining the depth loss function according to the depth image corresponding to the previous frame of sample color image and the depth image corresponding to the current frame of sample color image; and determining the photometric loss function according to the estimated pose when the image acquisition device acquires the current frame of sample color image and the depth image corresponding to the current frame of sample color image; and determining the IMU loss function according to the estimated pose when the image acquisition device acquires the current frame of sample color image and the sample IMU data corresponding to the previous frame of sample color image and the current frame of sample color image.
[0094] See also Fig.13 , Fig.13 It is a schematic diagram of the framework of an embodiment of the pose estimation device of the present application. In this embodiment, the pose estimation device 130 includes: a data acquisition module 1300, which is used to acquire a plurality of frames of continuous target color images using an image acquisition device, and determine the target IMU data corresponding to the image acquisition device when acquiring the plurality of frames of continuous target color images; a pose estimation module 1302, which is used to input the plurality of frames of continuous target color images and the corresponding target IMU data into a visual inertial odometer model, and obtain the estimated pose when the image acquisition device acquires the target color images; wherein the visual inertial odometer model is trained using any of the above-mentioned training methods for the visual inertial odometer model.
[0095] In the above scheme, the data acquisition module 1300 uses the image acquisition device to acquire several frames of continuous target color images, and determines the target IMU data corresponding to the image acquisition device when acquiring the several frames of continuous target color images. The pose estimation module 1302 obtains the estimated pose when the image acquisition device acquires the target color images by inputting the several frames of continuous target color images and the corresponding target IMU data into the visual inertial odometry model. Since the visual inertial odometry model is trained using the training method of the visual inertial odometry model of the first aspect, that is, the visual inertial odometry model integrates visual information and IMU information in the network, a more accurate and robust pose estimation result can be obtained by utilizing the respective advantages of the two.
[0096] See also Fig.14 , Figure 71 is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 140 includes a memory 141 and a processor 142 coupled to each other, and the processor 142 is used to execute program instructions stored in the memory 141 to implement any of the above-mentioned visual inertial odometer model training methods, or any of the above-mentioned pose estimation methods. In a specific implementation scenario, the electronic device 140 may include but is not limited to: a microcomputer, a server.
[0097] Specifically, the processor 142 is used to control itself and the memory 141 to implement the training method of any of the above-mentioned visual inertial odometer models, or the steps in any of the above-mentioned posture estimation method embodiments. The processor 142 may also be referred to as a CPU (Central Processing Unit). The processor 142 may be an integrated circuit chip having signal processing capabilities. The processor 142 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 142 may be implemented by an integrated circuit chip.
[0098] In the above scheme, the processor 142 obtains a sample image set and a sample IMU data set, wherein the sample image set includes a plurality of frames of continuous sample color images obtained by using an image acquisition device, and the sample IMU data set includes corresponding sample IMU data obtained when obtaining a plurality of frames of continuous sample color images. After the sample image set and the sample IMU data set are input into the visual inertial odometer model, the visual inertial odometer model can be used to estimate the scene depth and the pose of the image acquisition device. Specifically, two frames of depth images corresponding to two adjacent frames of sample color images and the estimated pose when the image acquisition device obtains two adjacent frames of sample color images can be output. Therefore, based on the two frames of depth images corresponding to the two adjacent frames of sample color images and the pose obtained by the image acquisition device, the scene depth and the pose of the image acquisition device can be estimated. The estimated pose of two adjacent frames of sample color images and all sample IMU data corresponding to the two adjacent frames of sample color images are taken to determine the target loss function of the visual inertial odometry model. Therefore, the visual information and IMU information are integrated in the network. By utilizing the advantages of both, a more accurate and robust visual inertial odometry model can be obtained. In addition, the visual inertial odometry is implemented using a deep learning framework. Compared with the traditional nonlinear method based on BA (Bundle-Adjustment), there is no need for complex initialization and iteration process, and the model is simpler. It solves the complex initialization and optimization problems in the traditional nonlinear optimization algorithm based on BA, and avoids tracking loss in complex scenes.
[0099] See also Fig.15 , Fig.15 The computer-readable storage medium 150 stores program instructions 1500 that can be executed by a processor, and the program instructions 1500 are used to implement the training method of any of the above-mentioned visual inertial odometer models, or the steps in any of the above-mentioned pose estimation method embodiments.
[0100] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0102] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
Claims
1. A training method for a visual inertial odometry model, characterized in that: include: Acquire a sample image set and a sample IMU data set; wherein the sample image set includes a plurality of frames of continuous sample color images acquired by an image acquisition device, and the sample IMU data set includes corresponding sample IMU data acquired when acquiring the plurality of frames of continuous sample color images; Input two adjacent frames of sample color images in the sample image set and the sample IMU data corresponding to the two adjacent frames of sample color images into the visual inertial odometer model, and output two frames of depth images corresponding to the two adjacent frames of sample color images and the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images; Determine the target loss function of the visual inertial odometer model based on the two frames of depth images corresponding to the two adjacent frames of sample color images, the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images, and the sample IMU data corresponding to the two adjacent frames of sample color images; The target loss function is used to adjust the network parameters of the visual inertial odometry model.
2. The training method of the visual inertial odometry model according to claim 1, characterized in that: The visual inertial odometry model includes a depth estimation network, a visual encoding network, an IMU encoding network and a visual inertial fusion network; The method comprises: inputting two adjacent frames of sample color images in the sample image set and the sample IMU data corresponding to the two adjacent frames of sample color images into the visual inertial odometer model, outputting two frames of depth images corresponding to the two adjacent frames of sample color images and the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images, including: Inputting a sample color image in the sample image set into the depth estimation network to obtain a depth image corresponding to the sample color image; and The previous frame sample color image and the current frame sample color image in the sample image set are superimposed and input into the visual coding network to obtain visual feature coding; Inputting the sample IMU data corresponding to the previous frame sample color image and the current frame sample color image into the IMU encoding network to obtain IMU feature encoding; The visual feature code and the IMU feature code are input into the visual-inertial fusion network to obtain an estimated pose when the image acquisition device acquires the current frame sample color image.
3. The training method of the visual inertial odometry model according to claim 2, characterized in that: The depth estimation network includes an encoder and a decoder connected to each other; The step of inputting a sample color image in the sample image set into the depth estimation network to obtain a depth image corresponding to the sample color image includes: The sample color image is input into the depth estimation network, the sample color image is transformed into a depth feature map using the downsampling layer of the encoder, and then the depth feature map is transformed into a depth image corresponding to the sample color image using the upsampling layer of the decoder.
4. The training method of the visual inertial odometry model according to claim 2, characterized in that: The visual-inertial fusion network adopts an attention mechanism, and the visual-inertial fusion network includes a feedforward neural network; The step of inputting the visual feature code and the IMU feature code into the visual-inertial fusion network to obtain an estimated pose when the image acquisition device acquires the current frame sample color image comprises: The visual feature coding and the IMU feature coding are weightedly fused through an attention mechanism to obtain an optimized feature coding; The optimized feature encoding is processed using a feedforward neural network to obtain an estimated posture when the image acquisition device acquires the current frame sample color image.
5. The training method of the visual inertial odometry model according to claim 4, characterized in that: The visual-inertial fusion network also includes a first multi-layer perceptron and a second multi-layer perceptron; The weighted fusion of the visual feature coding and the IMU feature coding by the attention mechanism to obtain the optimized feature coding includes: Inputting the IMU feature code into the first multilayer perceptron and the second multilayer perceptron respectively to obtain a plurality of key-value pairs, each of which includes a key and a value; The similarity between the visual feature code and the key in each key-value pair is obtained, the similarity is used as a weight, the weight is multiplied by the value in the corresponding key-value pair and then the sum is calculated to obtain the optimized feature code.
6. The training method of the visual inertial odometry model according to claim 2, characterized in that: The target loss function includes a depth loss function, a photometric loss function and an IMU loss function; The method of determining the target loss function of the visual inertial odometer model based on the two depth images corresponding to the two adjacent sample color images, the estimated pose when the image acquisition device acquires the two adjacent sample color images, and the sample IMU data corresponding to the two adjacent sample color images, includes: Determining the depth loss function according to the depth image corresponding to the previous frame sample color image and the depth image corresponding to the current frame sample color image; and Determining the photometric loss function according to the estimated pose when the image acquisition device acquires the current frame sample color image and the depth image corresponding to the current frame sample color image; and The IMU loss function is determined based on the estimated posture when the image acquisition device obtains the current frame sample color image and the sample IMU data corresponding to the previous frame sample color image and the current frame sample color image.
7. A method for posture estimation, characterized in that: The posture estimation method comprises: Acquire a plurality of frames of continuous target color images using an image acquisition device, and determine target IMU data corresponding to when the image acquisition device acquires the plurality of frames of continuous target color images; Inputting the plurality of frames of continuous target color images and corresponding target IMU data into a visual inertial odometer model to obtain an estimated position and posture when the image acquisition device acquires the target color image; Wherein, the visual inertial odometry model is trained using the visual inertial odometry model training method described in any one of claims 1 to 6.
8. A training device for a visual inertial odometry model, characterized in that: The training device of the visual inertial odometry model comprises: A sample acquisition module, used to acquire a sample image set and a sample IMU data set; wherein the sample image set includes a plurality of frames of continuous sample color images acquired by an image acquisition device, and the sample IMU data set includes sample IMU data corresponding to when the image acquisition device acquires the plurality of frames of continuous sample color images; A processing module, used for inputting two adjacent frames of sample color images in the sample image set and the sample IMU data corresponding to the two adjacent frames of sample color images into the visual inertial odometer model, and outputting two frames of depth images corresponding to the two adjacent frames of sample color images and the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images; A loss function determination module, used to determine the target loss function of the visual inertial odometer model based on the two frames of depth images corresponding to the two adjacent frames of sample color images, the estimated pose when the image acquisition device acquires the two adjacent frames of sample color images, and the sample IMU data corresponding to the two adjacent frames of sample color images; The parameter adjustment module is used to adjust the network parameters of the visual inertial odometry model using the target loss function.
9. A posture estimation device, characterized in that: The posture estimation device comprises: A data acquisition module, used to acquire a plurality of frames of continuous target color images by using an image acquisition device, and determine the target IMU data corresponding to when the image acquisition device acquires the plurality of frames of continuous target color images; A posture estimation module, used for inputting the plurality of frames of continuous target color images and corresponding target IMU data into a visual inertial odometer model to obtain an estimated posture when the image acquisition device acquires the target color image; Wherein, the visual inertial odometry model is trained using the visual inertial odometry model training method described in any one of claims 1 to 6.
10. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the training method of the visual inertial odometer model described in any one of claims 1 to 6, or the pose estimation method described in claim 7.
11. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by the processor, the training method of the visual inertial odometer model described in any one of claims 1 to 6, or the pose estimation method described in claim 7 is implemented.
Citation Information
Patent Citations
Gesture interaction method and device based on AR scene, storage medium and communication terminal
CN110221690A
Motion scene reconstruction unsupervised method based on IMU / monocular images
CN111311685A
