Position determination method, device, electronic device, and readable storage medium

Abstract

This application discloses a posture determination method, device, electronic device, and readable storage medium, belonging to the field of positioning technology. The posture determination method provided by this application includes: initializing a visual inertial positioning system based on image data of two adjacent frames to obtain the direction of gravity; determining the transformation relationship between the world coordinate system and the inertial coordinate system based on the gravity direction; and according to the transformation relationship, transforming the coordinates of the current frame image in the world coordinate system to the inertial coordinate system to obtain the first posture of the current frame image in the inertial coordinate system. The technical solution provided by this application can solve the problems of complex posture positioning methods and low positioning efficiency in related technologies.

Description

Technical Field

[0001] The present application belongs to the field of positioning technology, and specifically relates to a posture determination method, device, electronic device and readable storage medium. Background Art

[0002] Currently, scenario-based augmented reality (AR) visual inertial positioning is widely used in daily life, such as virtual shopping and advertising in large shopping malls, map navigation within large buildings like train stations and airports, and the fusion of virtual and real experiences at tourist attractions. Existing visual inertial positioning solutions mostly rely on map point cloud construction, relocalization, and tracking to achieve pose positioning. However, these solutions are computationally complex and have low positioning efficiency. Summary of the Invention

[0003] The embodiments of the present application provide a posture determination method, device, electronic device and readable storage medium, which can solve the problems of complex posture positioning methods and low positioning efficiency in related technologies.

[0004] In a first aspect, an embodiment of the present application provides a method for determining a posture, including:

[0005] Initialize the visual inertial positioning system based on the image data of two adjacent frames to obtain the gravity direction;

[0006] Determining a conversion relationship from a world coordinate system to an inertial coordinate system based on the gravity direction;

[0007] According to the conversion relationship, the coordinates of the current frame image in the world coordinate system are converted to the inertial coordinate system to obtain the first position of the current frame image in the inertial coordinate system.

[0008] In a second aspect, an embodiment of the present application provides a posture determination device, comprising:

[0009] An initialization module is used to initialize the visual inertial positioning system based on image data of two adjacent frames to obtain the direction of gravity;

[0010] A determination module, configured to determine a transformation relationship from a world coordinate system to an inertial coordinate system based on the gravity direction;

[0011] The conversion module is used to convert the coordinates of the current frame image in the world coordinate system to the inertial coordinate system according to the conversion relationship, so as to obtain the first position of the current frame image in the inertial coordinate system.

[0012] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the posture determination method described in the first aspect are implemented.

[0013] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the posture determination method as described in the first aspect are implemented.

[0014] In the embodiment of the present application, after determining the conversion relationship from the world coordinate system to the inertial coordinate system, the coordinates of the current frame image captured by the camera in the world coordinate system can be converted to the inertial coordinate system based on the conversion relationship, thereby obtaining the first pose of the current frame image in the inertial coordinate system. In this way, there is no need to achieve pose positioning through map point cloud construction, repositioning, etc., making the method for electronic devices to determine pose simpler and faster, effectively improving the efficiency of electronic devices in achieving pose positioning based on camera-captured images. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of a method for determining a posture provided by an embodiment of the present application;

[0016] Figure 2 is a schematic diagram of an inertial coordinate system involved in a posture determination method provided in an embodiment of the present application;

[0017] Figure 3 This is a flowchart of scene recognition in a posture determination method provided in an embodiment of the present application;

[0018] Figure 4 Schematic diagram of a visual common view relationship in a posture determination method provided in an embodiment of the present application;

[0019] Figure 5 is a structural diagram of a posture determination device provided in an embodiment of the present application;

[0020] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0022] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0023] The following describes in detail the posture determination method provided in the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0024] Please refer to Figure 1 , Figure 1 This is a flow chart of a method for determining a posture provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0025] Step 101: Initialize the visual inertial positioning system based on image data of two adjacent frames to obtain the direction of gravity.

[0026] It should be noted that the method provided in the embodiments of the present application can be applied to electronic devices, such as mobile phones, tablet computers, computers, smart wearable devices, etc. For better understanding, the posture determination method provided in the embodiments of the present application will be described below using electronic devices as the execution subject.

[0027] In the embodiment of the present application, the two adjacent frames of images may be any two adjacent frames of image frames captured by a camera (or also called a camera) of an electronic device.

[0028] Optionally, after starting the camera, the electronic device sequentially acquires image frames captured by the camera, and initializes a visual-inertial odometry (VIO) system based on the image data of two adjacent frames in the order in which the sequentially acquired image frames are acquired, until VIO initialization is successful. For example, if VIO initialization fails based on the image data of the first and second image frames acquired after the camera is started, VIO initialization continues based on the image data of the second and third image frames. If this fails, VIO initialization continues based on the image data of the third and fourth image frames until VIO initialization is successful.

[0029] It can be understood that the image captured by the camera (or camera) is a two-dimensional image of a plane, while the actual scene corresponding to the image captured by the camera is a three-dimensional space, for example, corresponding to the world coordinate system. Through the matching relationship between the image captured by the camera (two-dimensional) and the actual scene of the image (three-dimensional), the corresponding relationship from two-dimensional to three-dimensional can be obtained, and then the position of the image captured by the camera in the world coordinate system can be calculated. The visual inertial positioning system can realize the fusion of visual information and inertial information. The image captured by the camera also represents visual information. The image captured by the camera can be converted to the world coordinate system, and then the visual inertial positioning system can realize the conversion of the image captured by the camera from the world coordinate system to the inertial coordinate system. As Figure 2 As shown, the x-axis, y-axis, and z-axis of the inertial coordinate system correspond to the pitch angle (pitch), yaw angle (yaw), and roll angle (roll), respectively.

[0030] Step 102: Determine a transformation relationship from a world coordinate system to an inertial coordinate system based on the gravity direction.

[0031] It should be noted that the inertial coordinate system is related to the inertial measurement unit (IMU) data. The IMU data is used to measure the three-axis attitude angle and acceleration of an object. The acceleration in the IMU data is affected by gravity, and then the direction of gravity is determined to estimate the posture based on the IMU data.

[0032] In the embodiment of the present application, in the process of initializing the visual-inertial positioning system based on the image data of two adjacent frames, it is necessary to obtain the camera-captured images and IMU data to perform continuous feature tracking and pose calculation to complete the initialization of the visual-inertial positioning system. The pose calculation is related to the direction of gravity. Based on the initialization of the visual-inertial positioning system, the direction of gravity can be obtained, and the conversion relationship from the world coordinate system to the inertial coordinate system can be determined by the direction of gravity, thereby making it possible to more simply and quickly calculate the pose of the current frame image.

[0033] Step 103: According to the conversion relationship, the coordinates of the current frame image in the world coordinate system are converted to the inertial coordinate system to obtain the first position of the current frame image in the inertial coordinate system.

[0034] In the embodiment of the present application, after determining the conversion relationship from the world coordinate system to the inertial coordinate system, the coordinates of the current frame image captured by the camera in the world coordinate system can be converted to the inertial coordinate system based on the conversion relationship, thereby obtaining the first pose of the current frame image in the inertial coordinate system. In this way, there is no need to achieve pose positioning through map point cloud construction, repositioning, etc., making the method for electronic devices to determine pose simpler and faster, effectively improving the efficiency of electronic devices in achieving pose positioning based on camera-captured images.

[0035] Optionally, initializing the visual inertial positioning system based on image data of two adjacent frames of images to obtain the direction of gravity includes:

[0036] Obtain first IMU data and second IMU data corresponding to two adjacent frames of images respectively;

[0037] A visual inertial positioning system is initialized based on the first IMU data and the second IMU data to obtain the gravity direction.

[0038] For example, the two adjacent image frames include a first image frame and a second image frame, that is, first IMU data corresponding to the first image frame and second IMU data corresponding to the second image frame are obtained. Furthermore, by performing IMU pre-integration on the first IMU data and the second IMU data, data such as translation, velocity, and rotation can be obtained. In addition, parameters such as the position of the two adjacent image frames collected by the camera in the camera coordinate system to the world coordinate system can be obtained. Then, based on these parameters such as translation and velocity, the visual inertial positioning system is initialized to obtain the gravity direction, thereby achieving rapid positioning of the posture of the current frame image.

[0039] Optionally, initializing a visual inertial positioning system based on the first IMU data and the second IMU data to obtain the gravity direction includes:

[0040] Calculate the translation and velocity of the IMU pre-integration based on the first IMU data and the second IMU data;

[0041] Respectively obtaining positions of the two adjacent frames of images converted from the camera coordinate system to the world coordinate system, and obtaining a difference between the positions of the two adjacent frames of images converted from the world coordinate system;

[0042] Obtaining the rotation amount from the world coordinate system to the inertial coordinate system and the displacement of the camera coordinate system relative to the inertial coordinate system;

[0043] constructing an initialization matrix equation based on the translation amount, the speed, the difference, the rotation amount, and the displacement;

[0044] The visual inertial positioning system is initialized based on the initialization matrix equation to obtain the direction of gravity.

[0045] Specifically, after obtaining first IMU data and second IMU data corresponding to two adjacent frames of images, the translation amount and speed of the IMU pre-integration are calculated based on the first IMU data and the second IMU data.

[0046] Wherein, the two adjacent frames of images are images captured by the camera, and thus the coordinate positions of the two adjacent frames of images in the camera coordinate system can be obtained, and the two adjacent frames of images are converted from the camera coordinate system to the world coordinate system, and the difference in the coordinate positions of the two adjacent frames of images after conversion to the world coordinate system is obtained. For example, the two adjacent frames of images are the first image frame and the second image frame, that is, the first image frame is converted from the camera coordinate system to the world coordinate system, and the second image frame is converted from the camera coordinate system to the world coordinate system, and the difference between the coordinate position of the first image frame in the world coordinate system and the coordinate position of the second image frame in the world coordinate system is obtained. Wherein, the conversion of a feature point in the image frame from the camera coordinate system to the world coordinate system can be referred to the relevant technology, and this application will not elaborate on this.

[0047] In addition, it is also necessary to obtain the rotation amount from the world coordinate system to the inertial coordinate system, as well as the displacement of the camera coordinate system relative to the inertial coordinate system. The rotation amount from the world coordinate system to the inertial coordinate system can be obtained by referring to related technologies, and the displacement of the camera coordinate system relative to the inertial coordinate system can be determined by converting a feature point in the camera coordinate system to the inertial coordinate system based on the displacement of the feature point.

[0048] Furthermore, an initialization matrix equation is constructed based on the translation, velocity, difference, rotation and displacement obtained above, and the direction of gravity is obtained by solving the initialization matrix equation to achieve initialization of the visual inertial positioning system.

[0049] For example, construct the initialization matrix equation as follows:

[0050]

[0051] Among them, it is assumed that the two adjacent frames of images are two frames of images captured by the camera at time k and time k+1, and then the first IMU data corresponding to the image at time k and the second IMU data corresponding to time k+1 are obtained. The translation amount and speed of the IMU pre-integration are calculated based on the first IMU data and the second IMU data, that is, the translation amount in the above formula and speed and are the positions of the two frames of images from the camera coordinate system to the world coordinate system at time k and time k+1 respectively. is the displacement of the camera coordinate system relative to the inertial coordinate system, is the rotation of the image from the world coordinate system to the inertial coordinate system at time k, is the rotation of the image from the inertial coordinate system to the world coordinate system at time k+1, Δt is the time interval between time k and time k+1, I is the three-dimensional unit matrix, g w is the direction of gravity, is the speed at time k, is the velocity at time k+1.

[0052] Furthermore, through the above initialization matrix, the direction of gravity can be solved, so that the visual inertial positioning system can be initialized, and then the transformation relationship from the world coordinate system to the inertial coordinate system can be determined based on the direction of gravity, which is more conducive to quickly realizing the position positioning of the current frame image.

[0053] In the embodiment of the present application, the adjacent frame images include a first image frame, and determining the conversion relationship from the world coordinate system to the inertial coordinate system based on the gravity direction includes:

[0054] Acquire a second pose of the first image frame in a world coordinate system;

[0055] A transformation relationship from a world coordinate system to an inertial coordinate system is determined based on the gravity direction and the second posture.

[0056] It should be noted that the first image frame is an image captured by the camera. Based on the coordinates of the feature points in the first image frame in the camera coordinate system and the corresponding coordinates in the world coordinate system, the 3D-2D matching relationship of the image feature points can be obtained, thereby calculating the second posture of the first image frame in the world coordinate system. Based on the second posture and the gravity direction obtained after the visual inertial positioning system is initialized, the conversion relationship from the world coordinate system to the inertial coordinate system is determined.

[0057] The acquiring of the second pose of the first image frame in the world coordinate system includes:

[0058] Obtaining a first global descriptor of the first image frame;

[0059] Obtaining a second global descriptor of each frame of map image in the target map;

[0060] Obtaining a target second global descriptor having the greatest similarity to the first global descriptor, and determining a corresponding key map frame based on the target second global descriptor;

[0061] A matching relationship between the 3D feature points and the 2D feature points in the key map frame is obtained, and a second pose of the first image frame in the world coordinate system is determined according to the matching relationship.

[0062] In an embodiment of the present application, a global descriptor of the first image frame, i.e., the first global descriptor, is obtained. Simultaneously, a global descriptor, i.e., the second global descriptor, is extracted from the image in the target map using a deep network model. The target map may refer to a map including the scene in the first image frame, and the target map is a map image in a world coordinate system. Furthermore, the cosine similarity between the first global descriptor and the second global descriptor is calculated, and based on the maximum similarity, the target second global descriptor that is most similar to the first global descriptor is obtained. The map frame image corresponding to the target second global descriptor is also the map frame image that is most similar to the first image frame, and this most similar map frame image is then determined as the key map frame.

[0063] like Figure 3 As shown, the target map and the first image frame may be input into a feature extraction network, respectively, to obtain a second global descriptor and a first global descriptor, respectively, output by the feature extraction network. A similarity calculation is performed on the second global descriptor and the first global descriptor to obtain a target second global descriptor with the highest similarity to the first global descriptor. The map frame image corresponding to the target second global descriptor is then determined as a key frame image. The feature extraction network may be a pre-trained neural network model, and the training method of the neural network model may refer to related technologies, which will not be described in detail in this embodiment.

[0064] Among them, the key map frame image is a map frame image in the world coordinate system, and then the 3D feature points in the key map frame image are obtained, and the 2D feature points in the key map frame are obtained at the same time, and then the matching relationship between the 3D feature points and the 2D feature points in the key map frame can be determined; the 2D feature points in the first image frame are obtained, and based on the matching relationship between the 3D feature points and the 2D feature points, the 3D feature points in the first image frame can be obtained, and then the second posture of the first image frame in the world coordinate system is calculated based on the 3D feature points in the first image frame. Among them, the second posture includes the rotation parameter R cw and the translation parameter t cw .

[0065] In an embodiment of the present application, a similarity comparison is performed between the first global descriptor of the first image frame and the second global descriptor of the target map to obtain a key map frame that is most similar to the first image frame. The matching relationship between the 3D feature points and the 2D feature points is determined based on the key map frame. In this way, the second pose of the first image frame in the world coordinate system can be determined based on the matching relationship, making the determination of the pose of the first image frame more accurate.

[0066] It should be noted that the two adjacent frames of images are images captured by the electronic device from the camera, and are used in sequence in the order of capture to initialize the visual inertial positioning system until the visual inertial positioning system is successfully initialized. The first image frame can be any one of the two adjacent frames of images, or it can be the first frame of image that successfully initializes the visual inertial positioning system.

[0067] Furthermore, after determining the second posture of the first image frame, a conversion relationship from a world coordinate system to an inertial coordinate system is determined based on the second posture and the direction of gravity. Optionally, determining the conversion relationship from a world coordinate system to an inertial coordinate system based on the direction of gravity and the second posture includes:

[0068] Determining a conversion coefficient from a world coordinate system to an inertial coordinate system based on the gravity direction;

[0069] A conversion relationship from a world coordinate system to an inertial coordinate system is determined based on the conversion coefficient and the second posture.

[0070] In an embodiment of the present application, after initializing the visual inertial positioning system based on two adjacent frames of images and obtaining the direction of gravity, a conversion coefficient from the world coordinate system to the inertial coordinate system is obtained based on the direction of gravity. For example, the direction of gravity can be converted based on the Rodriguez formula to determine the conversion coefficient from the world coordinate system to the inertial coordinate system. Furthermore, the conversion relationship from the world coordinate system to the inertial coordinate system is determined based on the conversion coefficient and the second posture. For example, the conversion relationship from the world coordinate system to the inertial coordinate system can be determined based on the product of the conversion coefficient and the second posture.

[0071] Among them, the second posture is the posture of the first image frame in two adjacent frames of images. The transformation relationship from the world coordinate system to the inertial coordinate system is determined by the transformation relationship determined by the second posture and the gravity direction. It is also possible to establish a transformation relationship from the world coordinate system to the inertial coordinate system based on the posture of the image frame, and then the posture of subsequent image frames captured by the camera can also be determined based on the transformation relationship, making the positioning of the posture simpler and faster, and there is no need to achieve posture positioning through repositioning or other methods.

[0072] Optionally, determining a conversion coefficient from a world coordinate system to an inertial coordinate system based on the gravity direction includes:

[0073] Determining the gravitational acceleration in the world coordinate system and the gravitational acceleration in the inertial coordinate system based on the gravity direction;

[0074] Determining a rotation matrix between the world coordinate system and the inertial coordinate system according to the gravitational acceleration in the world coordinate system and the gravitational acceleration in the inertial coordinate system;

[0075] A conversion coefficient from the world coordinate system to the inertial coordinate system is determined according to the rotation matrix.

[0076] Specifically, after the direction of gravity is obtained based on the initialization of the visual inertial positioning system, for example, the direction of gravity is obtained based on the initialization matrix established in the above embodiment, the direction of gravity can be optimized to obtain the gravity in the image coordinate system, and the image coordinate system is the image coordinate system corresponding to the first image frame in the two adjacent frames of images; the gravity acceleration in the world coordinate system and the gravity acceleration in the inertial coordinate system are obtained, and then the rotation matrix between the world coordinate system and the inertial coordinate system is solved through the relationship between the gravity acceleration in the world coordinate system and the gravity acceleration in the inertial coordinate system. Based on the rotation matrix, the conversion coefficient from the world coordinate system to the inertial coordinate system can be obtained.

[0077] Furthermore, the conversion relationship from the world coordinate system to the inertial coordinate system is determined based on the conversion coefficient and the second posture of the first image frame. For example, the following formula can be obtained:

[0078]

[0079] in, Indicates the conversion coefficient from the world coordinate system to the inertial coordinate system, T c w represents the second pose of the first image frame, which is the pose of the first image frame in the world coordinate system. represents the pose of the first image frame in the inertial coordinate system. Based on the above formula, the transformation relationship from the world coordinate system to the inertial coordinate system can be obtained. That is, the pose of the image frame in the inertial coordinate system is the product of the transformation coefficient and the pose of the image frame in the world coordinate system.

[0080] It should be noted that

[0081] in, is the rotation parameter of the first image frame in the inertial coordinate system, is the translation parameter of the first image frame in the inertial coordinate system.

[0082] In an embodiment of the present application, after determining the conversion relationship from the world coordinate system to the inertial coordinate system, the coordinates of the current frame image captured by the camera in the world coordinate system can be converted to the inertial coordinate system to obtain the first position of the current frame image in the relational coordinate system.

[0083] For example, get the pose of the current frame image in the world coordinate system Based on the above conversion relationship, the following formula is obtained:

[0084]

[0085] in, is the first pose of the current frame image in the inertial coordinate system, is the conversion coefficient from the world coordinate system to the inertial coordinate system, is the pose of the current frame image in the world coordinate system.

[0086] In this way, by determining the transformation relationship from the world coordinate system to the inertial coordinate system, the method of determining the posture of the current frame image is made more convenient, and there is no need to determine the posture through repositioning or other methods, which effectively improves the efficiency of posture positioning.

[0087] Optionally, the method further includes:

[0088] Constructing a nonlinear least squares problem based on the current frame image and a frame image before the current frame image to optimize the pose of the current frame image to obtain an optimized third pose;

[0089] Furthermore, the step of converting the coordinates of the current frame image in the world coordinate system to the inertial coordinate system according to the conversion relationship to obtain the first pose of the current frame image in the inertial coordinate system includes:

[0090] According to the conversion relationship, the coordinates of the third posture in the world coordinate system are converted to the inertial coordinate system to obtain the first posture of the current frame image in the inertial coordinate system.

[0091] In an embodiment of the present application, the electronic device can optimize the pose of the current frame image captured by the camera based on constructing a nonlinear least squares problem. Optionally, an IMU constraint can be first constructed for the current frame image to obtain the relative pose relationship between the current frame image and the previous frame image, and the pose of the current frame image can be estimated based on this; a visual common view relationship between the current frame image and the previous frame image can be constructed to obtain common view landmark information, and then a nonlinear least squares problem can be constructed to optimize the relevant parameters of the current frame image to obtain the optimized pose of the current frame image.

[0092] It should be noted that for images continuously captured by a camera, two or more of these images may have a visual co-viewing relationship. Figure 4 As shown, the first four frames of images collected by the camera correspond to the first common viewpoint (i.e. Figure 4 The third and fourth frames correspond to the second common viewpoint, and the fourth and fifth frames correspond to the third common viewpoint. Based on the visual common view relationship between image frames, we can better formulate the nonlinear least squares problem between adjacent image frames.

[0093] Alternatively, a nonlinear least squares problem can be constructed based on the following formula:

[0094]

[0095] Where j is the current frame image, j=i+1, i.e. i is the previous frame image of the current frame image, V i,j represents the common view feature points between the i-th frame and the j-th frame, represents the visual constraint between the i-th frame and the j-th frame about the common viewpoint p, T i 、T j , ρ p Represent the pose of the i-th frame, the pose of the j-th frame, and the inverse depth of point p in the source frame, respectively. Represents the prior constraints at time 0, where M0 represents the IMU motion constraint at time 0, and C0 represents the visual (camera) constraint at time 0. Represents the IMU constraint between i and i+1, M i represents the motion constraint at time i, M i+1 represents the motion constraint at time i+1, ∑vis represents the visual constraint weight, ∑prior represents the prior constraint weight, and ∑imu represents the IMU constraint weight.

[0096] After constructing the nonlinear least squares problem based on the above formula, nonlinear optimization is performed to optimize the IMU data of the current frame image, and the optimized pose of the current frame image is calculated based on the optimized IMU data, that is, the third pose is obtained.

[0097] Furthermore, based on the conversion relationship from the world coordinate system to the inertial coordinate system, the coordinates of the third posture in the world coordinate system are converted to the inertial coordinate system, thereby obtaining the first posture of the current frame image in the inertial coordinate system.

[0098] In the embodiment of the present application, the conversion relationship is: the position of the image frame in the inertial coordinate system is the product of the conversion coefficient from the world coordinate system to the inertial coordinate system and the position of the image frame in the world coordinate system. Based on this conversion relationship, the first position of the current image frame can be calculated. Specifically, the first position of the current image frame can be expressed by the following formula:

[0099]

[0100] in, Indicates the conversion coefficient from the world coordinate system to the inertial coordinate system, represents the third pose of the current image frame, It represents the first pose of the first image frame in the inertial coordinate system, and then based on the above formula, the first pose of the current image frame converted to the inertial coordinate system can be obtained.

[0101] In the embodiment of the present application, for the current frame image captured by the camera, a nonlinear least squares problem is constructed for the current frame image and the previous frame image to optimize the pose of the current frame image. Based on the optimized pose, a conversion is performed from the world coordinate system to the inertial coordinate system to obtain the pose of the current frame image in the inertial coordinate system. In this way, by optimizing the pose of the current frame image, the accuracy of the pose positioning of the current frame image can be effectively improved.

[0102] Please refer to Figure 5 , Figure 5 This is a structural diagram of a posture determination device provided in an embodiment of the present application. Figure 5 As shown, the posture determination device 500 includes:

[0103] Initialization module 501, used to initialize the visual inertial positioning system based on image data of two adjacent frames of images to obtain the direction of gravity;

[0104] A determination module 502 is configured to determine a transformation relationship from a world coordinate system to an inertial coordinate system based on the gravity direction;

[0105] The conversion module 503 is used to convert the coordinates of the current frame image in the world coordinate system to the inertial coordinate system according to the conversion relationship, so as to obtain the first pose of the current frame image in the inertial coordinate system.

[0106] Optionally, the initialization module 501 is further configured to:

[0107] Obtaining first inertial measurement unit (IMU) data and second IMU data corresponding to two adjacent frames of image respectively;

[0108] A visual inertial positioning system is initialized based on the first IMU data and the second IMU data to obtain the gravity direction.

[0109] Optionally, the initialization module 501 is further configured to:

[0110] Calculate the translation and velocity of the IMU pre-integration based on the first IMU data and the second IMU data;

[0111] Respectively obtaining positions of the two adjacent frames of images converted from the camera coordinate system to the world coordinate system, and obtaining a difference between the positions of the two adjacent frames of images converted from the world coordinate system;

[0112] Obtaining the rotation amount from the world coordinate system to the inertial coordinate system and the displacement of the camera coordinate system relative to the inertial coordinate system;

[0113] constructing an initialization matrix equation based on the translation amount, the speed, the difference, the rotation amount, and the displacement;

[0114] The visual inertial positioning system is initialized based on the initialization matrix equation to obtain the gravity direction.

[0115] Optionally, the two adjacent image frames include a first image frame, and the determining module 502 includes:

[0116] an acquiring unit, configured to acquire a second pose of the first image frame in a world coordinate system;

[0117] A determination unit is used to determine a transformation relationship from a world coordinate system to an inertial coordinate system based on the gravity direction and the second posture.

[0118] Optionally, the acquiring unit is further configured to:

[0119] Obtaining a first global descriptor of the first image frame;

[0120] Obtaining a second global descriptor of each frame of map image in the target map;

[0121] Obtaining a target second global descriptor having the greatest similarity to the first global descriptor, and determining a corresponding key map frame based on the target second global descriptor;

[0122] A matching relationship between the 3D feature points and the 2D feature points in the key map frame is obtained, and a second pose of the first image frame in the world coordinate system is determined according to the matching relationship.

[0123] Optionally, the determining unit is further configured to:

[0124] Determining a conversion coefficient from a world coordinate system to an inertial coordinate system based on the gravity direction;

[0125] A conversion relationship from a world coordinate system to an inertial coordinate system is determined based on the conversion coefficient and the second posture.

[0126] Optionally, the determining unit is further configured to:

[0127] Determining the gravitational acceleration in the world coordinate system and the gravitational acceleration in the inertial coordinate system based on the gravity direction;

[0128] Determining a rotation matrix between the world coordinate system and the inertial coordinate system according to the gravitational acceleration in the world coordinate system and the gravitational acceleration in the inertial coordinate system;

[0129] A conversion coefficient from the world coordinate system to the inertial coordinate system is determined according to the rotation matrix.

[0130] Optionally, the device further comprises:

[0131] an optimization module, configured to construct a nonlinear least squares problem based on the current frame image and a frame image preceding the current frame image, so as to optimize the pose of the current frame image and obtain an optimized third pose;

[0132] The conversion module 503 is further configured to:

[0133] According to the conversion relationship, the coordinates of the third posture in the world coordinate system are converted to the inertial coordinate system to obtain the first posture of the current frame image in the inertial coordinate system.

[0134] In the embodiment of the present application, after determining the conversion relationship from the world coordinate system to the inertial coordinate system, the device can then, for the current frame image captured by the camera, convert the coordinates of the current frame image in the world coordinate system to the inertial coordinate system based on the conversion relationship, thereby obtaining the first pose of the current frame image in the inertial coordinate system. In this way, there is no need to achieve pose positioning through map point cloud construction, repositioning, etc., making the pose determination method simpler and faster, and effectively improving the efficiency of the device in achieving pose positioning based on camera-captured images.

[0135] The posture determination device 500 in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the embodiment of the present application does not specifically limit it.

[0136] The posture determination device 500 in the embodiment of the present application can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0137] The posture determination device 500 provided in the embodiment of the present application can achieve Figure 1To avoid repetition, the various processes implemented in the method embodiment will not be described here.

[0138] The present application also provides an electronic device. Figure 6 , Figure 6 This is a structural diagram of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device includes: a processor 600, a memory 620, and a program or instruction stored in the memory 620 and executable on the processor 600, the processor 600 being used to read the program or instruction in the memory 620; the electronic device also includes a bus interface and a transceiver 610.

[0139] The transceiver 610 is configured to receive and send data under the control of the processor 600 .

[0140] Among them, Figure 6 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 600 and memory represented by memory 620. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are all well known in the art and, therefore, will not be described further herein. The bus interface provides an interface. The transceiver 610 may be a plurality of components, i.e., a transmitter and a transceiver, providing a unit for communicating with various other devices on a transmission medium. The processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 may store data used by the processor 600 when performing operations.

[0141] The processor 600 is configured to read the program or instructions in the memory 620 and execute the following steps:

[0142] Initialize the visual inertial positioning system based on the image data of two adjacent frames to obtain the gravity direction;

[0143] Determining a conversion relationship from a world coordinate system to an inertial coordinate system based on the gravity direction;

[0144] According to the conversion relationship, the coordinates of the current frame image in the world coordinate system are converted to the inertial coordinate system to obtain the first position of the current frame image in the inertial coordinate system.

[0145] Optionally, the processor 600 is further configured to read a program or instruction in the memory 620 and perform the following steps:

[0146] Obtaining first inertial measurement unit (IMU) data and second IMU data corresponding to two adjacent frames of image respectively;

[0147] A visual inertial positioning system is initialized based on the first IMU data and the second IMU data to obtain the gravity direction.

[0148] Optionally, the processor 600 is further configured to read a program or instruction in the memory 620 and perform the following steps:

[0149] Calculate the translation and velocity of the IMU pre-integration based on the first IMU data and the second IMU data;

[0150] Respectively obtaining positions of the two adjacent frames of images converted from the camera coordinate system to the world coordinate system, and obtaining a difference between the positions of the two adjacent frames of images converted from the world coordinate system;

[0151] Obtaining the rotation amount from the world coordinate system to the inertial coordinate system and the displacement of the camera coordinate system relative to the inertial coordinate system;

[0152] constructing an initialization matrix equation based on the translation amount, the speed, the difference, the rotation amount, and the displacement;

[0153] The visual inertial positioning system is initialized based on the initialization matrix equation to obtain the gravity direction.

[0154] Optionally, the two adjacent image frames include a first image frame, and the processor 600 is further configured to read a program or instruction in the memory 620 and perform the following steps:

[0155] Acquire a second pose of the first image frame in a world coordinate system;

[0156] A transformation relationship from a world coordinate system to an inertial coordinate system is determined based on the gravity direction and the second posture.

[0157] Optionally, the processor 600 is further configured to read a program or instruction in the memory 620 and perform the following steps:

[0158] Obtaining a first global descriptor of the first image frame;

[0159] Obtaining a second global descriptor of each frame of map image in the target map;

[0160] Obtaining a target second global descriptor having the greatest similarity to the first global descriptor, and determining a corresponding key map frame based on the target second global descriptor;

[0161] A matching relationship between the 3D feature points and the 2D feature points in the key map frame is obtained, and a second pose of the first image frame in the world coordinate system is determined according to the matching relationship.

[0162] Optionally, the processor 600 is further configured to read a program or instruction in the memory 620 and perform the following steps:

[0163] Determining a conversion coefficient from a world coordinate system to an inertial coordinate system based on the gravity direction;

[0164] A conversion relationship from a world coordinate system to an inertial coordinate system is determined based on the conversion coefficient and the second posture.

[0165] Optionally, the processor 600 is further configured to read a program or instruction in the memory 620 and perform the following steps:

[0166] Determining the gravitational acceleration in the world coordinate system and the gravitational acceleration in the inertial coordinate system based on the gravity direction;

[0167] Determining a rotation matrix between the world coordinate system and the inertial coordinate system according to the gravitational acceleration in the world coordinate system and the gravitational acceleration in the inertial coordinate system;

[0168] A conversion coefficient from the world coordinate system to the inertial coordinate system is determined according to the rotation matrix.

[0169] Optionally, the processor 600 is further configured to read a program or instruction in the memory 620 and perform the following steps:

[0170] Constructing a nonlinear least squares problem based on the current frame image and a frame image before the current frame image to optimize the pose of the current frame image to obtain an optimized third pose;

[0171] According to the conversion relationship, the coordinates of the third posture in the world coordinate system are converted to the inertial coordinate system to obtain the first posture of the current frame image in the inertial coordinate system.

[0172] In the embodiment of the present application, after determining the conversion relationship from the world coordinate system to the inertial coordinate system, the coordinates of the current frame image captured by the camera in the world coordinate system can be converted to the inertial coordinate system based on the conversion relationship, thereby obtaining the first pose of the current frame image in the inertial coordinate system. In this way, there is no need to achieve pose positioning through map point cloud construction, repositioning, etc., making the method for electronic devices to determine pose simpler and faster, effectively improving the efficiency of electronic devices in achieving pose positioning based on camera-captured images.

[0173] The embodiment of the present application also provides a readable storage medium on which a program or instruction is stored, and when the program or instruction is executed by the processor, the above Figure 1 The various processes of the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0174] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0175] The embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the above Figure 1 The various processes of the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0176] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0177] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0178] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0179] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for determining a posture, characterized in that: include: Initializing a visual inertial positioning system based on image data of two adjacent frames of images to obtain a gravity direction, the initializing the visual inertial positioning system based on image data of two adjacent frames of images to obtain a gravity direction includes: obtaining first inertial measurement unit (IMU) data and second IMU data corresponding to the two adjacent frames of images respectively; initializing the visual inertial positioning system based on the first IMU data and the second IMU data to obtain the gravity direction, the initializing the visual inertial positioning system based on the first IMU data and the second IMU data to obtain the gravity direction includes: calculating the translation and velocity of the IMU pre-integration based on the first IMU data and the second IMU data; respectively obtaining positions of the two adjacent frames of images converted from the camera coordinate system to the world coordinate system, and obtaining the difference in positions of the two adjacent frames of images converted to the world coordinate system; obtaining a rotation from the world coordinate system to the inertial coordinate system and a displacement of the camera coordinate system relative to the inertial coordinate system; constructing an initialization matrix equation based on the translation, the velocity, the difference, the rotation and the displacement; initializing the visual inertial positioning system based on the initialization matrix equation to obtain a gravity direction; Determining a conversion relationship from a world coordinate system to an inertial coordinate system based on the gravity direction; According to the conversion relationship, the coordinates of the current frame image in the world coordinate system are converted to the inertial coordinate system to obtain the first position of the current frame image in the inertial coordinate system.

2. The method according to claim 1, characterized in that The two adjacent image frames include a first image frame, and determining a conversion relationship from a world coordinate system to an inertial coordinate system based on the gravity direction includes: Acquire a second pose of the first image frame in a world coordinate system; A transformation relationship from a world coordinate system to an inertial coordinate system is determined based on the gravity direction and the second posture.

3. The method according to claim 2, characterized in that The acquiring of a second pose of the first image frame in a world coordinate system includes: Obtaining a first global descriptor of the first image frame; Obtaining a second global descriptor of each frame of map image in the target map; Obtaining a target second global descriptor having the greatest similarity to the first global descriptor, and determining a corresponding key map frame based on the target second global descriptor; A matching relationship between the 3D feature points and the 2D feature points in the key map frame is obtained, and a second pose of the first image frame in the world coordinate system is determined according to the matching relationship.

4. The method according to claim 2, characterized in that The determining of a conversion relationship from a world coordinate system to an inertial coordinate system based on the gravity direction and the second posture includes: Determining a conversion coefficient from a world coordinate system to an inertial coordinate system based on the gravity direction; A conversion relationship from a world coordinate system to an inertial coordinate system is determined based on the conversion coefficient and the second posture.

5. The method according to claim 2, characterized in that The determining of the conversion coefficient from the world coordinate system to the inertial coordinate system based on the gravity direction includes: Determining the gravitational acceleration in the world coordinate system and the gravitational acceleration in the inertial coordinate system based on the gravity direction; Determining a rotation matrix between the world coordinate system and the inertial coordinate system according to the gravitational acceleration in the world coordinate system and the gravitational acceleration in the inertial coordinate system; A conversion coefficient from the world coordinate system to the inertial coordinate system is determined according to the rotation matrix.

6. The method according to claim 1, characterized in that The method further comprises: Constructing a nonlinear least squares problem based on the current frame image and a frame image before the current frame image to optimize the pose of the current frame image to obtain an optimized third pose; The step of converting the coordinates of the current frame image in the world coordinate system to the inertial coordinate system according to the conversion relationship to obtain the first pose of the current frame image in the inertial coordinate system includes: According to the conversion relationship, the coordinates of the third posture in the world coordinate system are converted to the inertial coordinate system to obtain the first posture of the current frame image in the inertial coordinate system.

7. A posture determination device, characterized in that: include: An initialization module is used to initialize the visual inertial positioning system based on the image data of two adjacent frames of images to obtain the direction of gravity. The initialization module is also used to: obtain the first inertial measurement unit IMU data and the second IMU data corresponding to the two adjacent frames of images respectively; initialize the visual inertial positioning system based on the first IMU data and the second IMU data to obtain the direction of gravity. The initialization module is also used to: calculate the translation and speed of the IMU pre-integration based on the first IMU data and the second IMU data; respectively obtain the positions of the two adjacent frames of images converted from the camera coordinate system to the world coordinate system, and obtain the difference in the positions of the two adjacent frames of images converted to the world coordinate system; obtain the rotation from the world coordinate system to the inertial coordinate system and the displacement of the camera coordinate system relative to the inertial coordinate system; and construct an initialization matrix equation based on the translation, the speed, the difference, the rotation and the displacement; Initializing the visual inertial positioning system based on the initialization matrix equation to obtain a gravity direction; A determination module, configured to determine a transformation relationship from a world coordinate system to an inertial coordinate system based on the gravity direction; The conversion module is used to convert the coordinates of the current frame image in the world coordinate system to the inertial coordinate system according to the conversion relationship, so as to obtain the first position of the current frame image in the inertial coordinate system.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the posture determination method according to any one of claims 1 to 6 are implemented.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the posture determination method according to any one of claims 1 to 6 are implemented.

Images