Pose determination method, apparatus, device, storage medium and program product

Abstract

The present disclosure relates to a pose information confirmation method, device, equipment, storage medium and program product, the method comprising: acquiring camera frames in a sliding window; determining to-be-optimized pose information corresponding to the camera frames; acquiring issued pose information corresponding to key frames in the sliding window; determining prior constraint conditions based on the issued pose information and the to-be-optimized pose; performing sliding window optimization on the to-be-optimized pose information based on the prior constraint conditions to obtain target pose information corresponding to the camera frames. The embodiment of the present disclosure incorporates the issued pose information corresponding to the key frames in the prior constraint conditions, thereby solving the problem of transmission delay.

Description

Technical Field

[0001] This disclosure relates to the field of positioning processing technology, and in particular to a pose determination method, apparatus, device, storage medium, and program product. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) refers to the process by which electronic devices, in an unknown environment, acquire visual images of their surroundings to build a map and achieve self-localization. With the development of science and technology, SLAM has important applications in mobile device localization, such as in mobile robots, AR (Augmented Reality), VR (Virtual Reality), drones, virtual visual positioning systems, mobile smart terminals, and wearable devices.

[0003] Localization fusion has always been an important topic in the practical application of SLAM, especially in mobile devices such as smartphones and wearable devices. When using SLAM algorithms for localization fusion, the moving window algorithm is often used for optimization. Mobile devices have higher requirements for the real-time performance of the algorithm, especially for wearable devices, which may experience drastic state changes in a short period of time. As a result, the number of keyframes included in the moving window used for optimization in the design of visual inertial odometry (VIO, used for real-time mapping and localization) schemes on mobile devices is very small.

[0004] When the sliding window contains very few keyframes, using keyframes for localization fusion will result in a long localization delay. Summary of the Invention

[0005] To address the aforementioned technical problems, embodiments of this disclosure provide a pose determination method, apparatus, device, storage medium, and program product. Embodiments of this disclosure incorporate the transmitted pose information corresponding to key frames into the prior constraints, thereby solving the problem of transmission delay.

[0006] In a first aspect, embodiments of this disclosure provide a pose determination method, the method being applied to a visual inertial odometry system, comprising:

[0007] Capture camera frames within the sliding window;

[0008] Determine the pose information to be optimized corresponding to the camera frame;

[0009] Obtain the downlink pose information corresponding to the keyframes in the sliding window;

[0010] Based on the issued pose information and the pose to be optimized, determine the prior constraints.

[0011] Based on the prior constraints, the pose information to be optimized is optimized by sliding window to obtain the target pose information corresponding to the camera frame.

[0012] Secondly, embodiments of this disclosure provide a pose determination device, the device being configured in a positioning system including a visual inertial odometry system, comprising:

[0013] The camera frame acquisition module is used to acquire camera frames that are in the sliding window;

[0014] The pose determination module is used to determine the pose information to be optimized corresponding to the camera frame;

[0015] The pose information acquisition module is used to acquire the pose information corresponding to the keyframe in the sliding window;

[0016] The constraint determination module is used to determine prior constraints based on the transmitted pose information and the pose to be optimized.

[0017] The target pose determination module is used to perform sliding window optimization on the pose information to be optimized based on the prior constraints to obtain the target pose information corresponding to the camera frame.

[0018] Thirdly, embodiments of this disclosure provide an electronic device, the electronic device comprising:

[0019] One or more processors;

[0020] Storage device for storing one or more programs;

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement the pose determination method as described in any of the first aspects above.

[0022] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the pose determination method as described in any one of the first aspects above.

[0023] Fifthly, embodiments of this disclosure provide a computer program product comprising a computer program or instructions that, when executed by a processor, implement the pose determination method as described in any one of the first aspects above.

[0024] This disclosure provides a pose determination method, apparatus, device, storage medium, and program product. The method includes: acquiring a camera frame in a sliding window; determining the pose information to be optimized corresponding to the camera frame; acquiring the transmitted pose information corresponding to a keyframe in the sliding window; determining prior constraints based on the transmitted pose information and the pose to be optimized; and performing sliding window optimization on the pose information to be optimized based on the prior constraints to obtain the target pose information corresponding to the camera frame. This disclosure solves the transmission latency problem by incorporating the transmitted pose information corresponding to the keyframe into the prior constraints. Attached Figure Description

[0025] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0026] Figure 1 This is a flowchart illustrating a pose determination method according to an embodiment of the present disclosure;

[0027] Figure 2 This is a schematic diagram of the structure of a pose determination device according to an embodiment of the present disclosure;

[0028] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0029] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0030] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0031] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0032] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0033] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0034] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0035] Currently, the mainstream positioning fusion methods mainly include: multi-sensor-based positioning fusion and prior map-based positioning fusion.

[0036] In the main approaches to multi-sensor-based positioning fusion, GPS data is combined with LiDAR used in autonomous driving to achieve fused positioning in outdoor scenarios. Mobile devices typically use visual-inertial odometry (VIO) solutions to achieve fused positioning, which combines a camera and an inertial measurement unit (IMU). In indoor scenarios, signal transmitters, such as Bluetooth, Wi-Fi, or special markers (reflective strips or QR codes), are usually placed in known locations to provide additional positioning information.

[0037] Multi-sensor-based positioning fusion solutions have high requirements for sensors and scenarios, making them difficult to apply on a large scale, especially on mobile devices where resources and costs are high.

[0038] The prior map-based localization fusion method is mainly used to build a prior map in similar scenarios, and then use this map to obtain the prior localization result when used again, and send the localization result to the real-time running system to improve the overall localization accuracy.

[0039] Location fusion methods based on prior maps require a prior map, thus necessitating the construction of an offline map first. The main problem is that map-based positioning is time-consuming, including information transmission and pose calculation. This time consumption is not ideal for real-time solutions on some mobile devices.

[0040] In this disclosure, key names that may appear below will be briefly introduced.

[0041] Simultaneous Localization and Mapping (SLAM) is a cutting-edge direction in spatial localization technology within the vision field. It primarily addresses the localization and map building problems of robots moving in unknown environments. In Virtual Reality (VR) or Augmented Reality (AR), SLAM-generated maps and the current viewpoint are used to overlay virtual objects, enhancing their realism. In the field of unmanned aerial vehicles (UAVs), SLAM constructs local maps to assist UAVs in autonomous obstacle avoidance. SLAM technology covers a very wide range. Based on different sensors, it can be categorized into 2D / 3D SLAM based on LiDAR, RGBD SLAM based on depth cameras, visual SLAM based on visual sensors, and VIO (Visual Inertial Odometry) based on visual sensors and inertial units.

[0042] Visual inertial odometry (VIO) integrates camera and inertial motion unit (IMU) data to achieve simultaneous localization and mapping (SLAM) algorithms. The camera and IMU are highly complementary. In VIO, the true scale of the camera trajectory can be estimated by aligning the pose estimated by the IMU with that estimated by the camera. Furthermore, the IMU can predict the pose of image frames and the position of feature points from the previous moment in the next frame, improving the matching speed of the feature tracking algorithm and its robustness against rapid rotation. Finally, the gravity vector provided by the accelerometer in the IMU can convert the estimated position into a true 3D spatial coordinate system.

[0043] The localization process of Visual Inertial Odometry (VIO) mainly includes five parts: image and IMU data preprocessing, initialization, local nonlinear optimization, loop closure detection, and global optimization. Image and IMU data preprocessing: For images, feature points are extracted, and optical flow tracking is performed using the KLT pyramid to prepare for subsequent visual-only initialization to solve for camera pose. For the IMU, the IMU data is pre-integrated to obtain the pose, velocity, and rotation angle at the current moment. Simultaneously, the pre-integration increment between adjacent frames, as well as the covariance and Jacobian matrices of the pre-integration, which will be used in the back-end optimization, are calculated. Initialization: In initialization, visual-only initialization is performed first to solve for the relative pose of the camera; then, the initialization parameters are solved by aligning with the IMU pre-integration. Local nonlinear optimization: Visual-inertial nonlinear optimization of the sliding window is performed, that is, optimizing visual constraints and IMU constraints within a large objective function. Local optimization refers to optimizing the variables within the window of the current camera frame and the previous n frames; local nonlinear optimization outputs a more accurate pose. Loop closure detection: Loop closure detection saves previously detected keyframes from the image. When returning to the same location, it uses feature point matching to determine if the location has been visited before. Global optimization: Global optimization, when loop closure occurs, utilizes camera constraints, IMU constraints, and loop closure detection constraints to perform non-linear optimization. Global optimization builds upon local optimization, outputting a more accurate pose.

[0044] It should be noted that the pose determination method provided in this disclosure is mainly applied to the local nonlinear optimization part of visual inertial odometry (VIO), which is the optimization of camera frames within a sliding window. In the embodiments of this disclosure, the camera frame obtained within the sliding window refers to the current camera frame within the sliding window. The target pose information corresponding to the camera frame refers to the relatively accurate pose information output by the local nonlinear optimization.

[0045] The pose determination method proposed in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0046] Figure 1 This is a flowchart of a pose determination method in an embodiment of the present disclosure. This embodiment can be applied to the case of local pose optimization in visual inertial odometry (VIO). The method can be executed by a pose determination device, which can be implemented in software and / or hardware and can be configured in an electronic device.

[0047] For example, the electronic device may be a mobile terminal, a fixed terminal, or a portable terminal, such as a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof.

[0048] For example, the electronic device can be a server, which can be a physical server or a cloud server. The server can be a single server or a server cluster.

[0049] like Figure 1 As shown, the pose determination method provided in this embodiment mainly includes steps S101-S105.

[0050] S101. Obtain the camera frame in the sliding window.

[0051] In the field of video, videos can be considered as a series of images that change continuously over time. A frame refers to each image, and a camera frame refers to a specific image captured by the camera of an electronic device.

[0052] Specifically, in the process of simultaneous localization and mapping (SLAM), as the number of keyframes and landmarks increases, the back-end bundle adjustment (BA) model will continuously increase, the computational load of the model will also continuously increase, and the computational efficiency will continuously decrease. To avoid this situation, a sliding window is used to limit the number of camera frames to be optimized to a certain extent, thereby controlling the scale of the bundle adjustment (BA) model.

[0053] The sliding window described above can be a time-domain based sliding window, a frequency-domain based sliding window, or a combination of time-domain and frequency-domain sliding windows. This embodiment does not limit the method of determining the sliding window.

[0054] In this embodiment, a time-domain based sliding window is used as an example. A time-domain based sliding window can be understood as a sliding window that moves forward continuously over time. The length of the sliding window does not change over time. For example, if a sliding window contains 20 image frames, after the camera of the electronic device acquires a new image frame, the new image frame is added to the sliding window, and the earliest acquired image frame in the window is discarded. This discarding of image frames is not a literal deletion, as directly discarding variables would result in information loss. Instead, these image frames are added to the edge optimization conditions of the sliding window to avoid information loss.

[0055] Specifically, in this embodiment of the disclosure, a camera frame in the sliding window can be understood as the latest image frame captured by the camera in the electronic device and added to the sliding window. Obtaining a camera frame in the sliding window can be understood as obtaining the latest image frame in the sliding window.

[0056] S102. Determine the pose information to be optimized corresponding to the camera frame.

[0057] The pose information includes the camera's position coordinates, height, and orientation information. The pose information to be optimized can be understood as the pose information corresponding to the camera frame before sliding window optimization.

[0058] The camera's position coordinates can be its two-dimensional coordinates in the world coordinate system, i.e., its coordinates in the Z-plane. The camera's height can be its height value in the world coordinate system, i.e., its height above the Z-plane. The orientation information can be the camera's orientation information in the world coordinate system. Furthermore, in this embodiment, the camera's position coordinates can be obtained using positioning technology, and the camera's orientation information can be obtained using an orientation meter or gyroscope, etc.

[0059] S103. Obtain the pose information corresponding to the key frame in the sliding window.

[0060] In this context, a keyframe within a sliding window can be any image frame within the window. Specifically, a keyframe refers to the frame in an animation or video where a character or object's movement is a key action. The determination of a keyframe is based on the data correlation between the oldest keyframe and the previous frame within the sliding window. It should be noted that this disclosure does not impose a time requirement on the oldest keyframe; it can be relatively close or far apart in time. However, the selection criteria for keyframes must meet certain conditions: the number of frames from the previous keyframe, the spatial distance from the nearest keyframe, and tracking quality. In other words, the quality of the keyframe must be sufficiently high, as it serves as the skeleton of Simultaneous Localization and Mapping (SLAM).

[0061] In one embodiment of this disclosure, in sliding window optimization, the first frame in the sliding window is used as the key frame. The first frame in the sliding window can refer to the image frame with the earliest acquisition time in the sliding window.

[0062] The transmitted pose information corresponding to the keyframe can be understood as the pose information obtained after processing the local pose information corresponding to the keyframe. Specifically, the local pose information corresponding to the keyframe refers to the pose information determined in the keyframe that does not require processing.

[0063] In one embodiment of this disclosure, the pose information obtained after processing the local pose information corresponding to the key frame can be executed by the electronic device itself, or the electronic device can upload the local pose information corresponding to the key frame to the positioning server, and the positioning server can process the local pose information corresponding to the key frame to obtain the sent pose information corresponding to the key frame.

[0064] In one embodiment of this disclosure, the method further includes: determining the first frame in the sliding window as a key frame; sending the local pose information corresponding to the key frame to the positioning server, so that the positioning server processes the local pose information to obtain the sent pose information corresponding to the key frame.

[0065] First, it should be noted that the location server can be a local server or a cloud server. This embodiment does not impose any specific limitations. Specifically, this embodiment requires a new thread to implement the location service. The location server is named M_LOC, and M_LOC can be a module in the cloud, primarily used to implement the location function; it can only run one task at a time.

[0066] Furthermore, before each sliding window optimization, a status query request is sent to the positioning server. This status query request instructs the positioning server M_LOC to return its local working status. If the working status returned by the positioning server M_LOC is busy, the local subsequent process continues, i.e., the original local sliding window optimization process. If the working status returned by the positioning server M_LOC is idle, the local pose information T corresponding to the keyframe is... local_wi Uploaded to the positioning server M_LOC, so that the positioning server M_LOC can obtain the local pose information T corresponding to the keyframe. local_wi The process is performed to obtain the pose information corresponding to the keyframes.

[0067] It should be noted that the location server M_LOC needs to communicate with multiple electronic devices to provide location services for them. The location server M_LOC is in a busy state, which can be understood as the location server M_LOC being performing the task of providing location services for other electronic devices, or performing other tasks of its own.

[0068] In this embodiment of the disclosure, the local pose information corresponding to the key frame is uploaded to the positioning server, and the positioning server processes the local pose information corresponding to the key frame to obtain the corresponding downlink pose information. In this way, the amount of local computation of the electronic device can be reduced.

[0069] In one embodiment of this disclosure, the positioning server processes the local pose information to obtain the transmitted pose information corresponding to the keyframe, including: if the local pose information is image data, the positioning server processes the image data based on the map data stored on the local end and the PnP algorithm to obtain the transmitted pose information corresponding to the keyframe.

[0070] Specifically, PnP (Perspective-n-Points) refers to the object motion localization problem from 3D to 2D point pairs. That is, given the coordinates of the object in the world coordinate system and the pixel coordinates of the object in the image plane of the camera, the pose information of the camera is calculated. In the specific calculation, the pose information can only be obtained when n is greater than 4.

[0071] The positioning server described in this embodiment stores a cloud map. The positioning server receives image data sent from the electronic device, processes the image data based on the PnP algorithm and the cloud map, and obtains the downlink pose information corresponding to the key frame.

[0072] In one embodiment of this disclosure, the positioning server processes the local pose information to obtain the transmission pose information corresponding to the key frame, including: if the local pose information is a pose matrix, the positioning server obtains the timestamp corresponding to the pose matrix; the positioning server obtains the positioning information corresponding to the Global Positioning System based on the timestamp corresponding to the pose matrix, and uses it as the transmission pose information corresponding to the key frame.

[0073] The positioning information corresponding to the ball positioning system can be one or more of the GPS positioning system, Beidou positioning and navigation system, etc.

[0074] In this embodiment of the disclosure, the positioning server reads the timestamp corresponding to the pose matrix and obtains the positioning information collected by the timestamp in the global positioning system, and uses the positioning information as the pose information to be sent to the key frame.

[0075] In one embodiment of this disclosure, obtaining the transmission pose information corresponding to the keyframe in the sliding window includes: sending a pose query request to a positioning server, wherein the pose query request is used to instruct the positioning server to query whether the transmission pose information exists on the local end; if the transmission pose information exists in the positioning server, then obtaining the transmission pose information corresponding to the keyframe in the sliding window from the positioning server.

[0076] Specifically, the positioning server can be a cloud server whose main function is to achieve visual positioning. It can only run one task at a time, and when the positioning server is in operation, it cannot fulfill positioning requests from other devices. The transmitted pose information is fused with the local pose information corresponding to the keyframes.

[0077] Before each sliding window optimization, the electronic device sends a pose query request to the positioning server. This request instructs the positioning server to check if the transmitted pose information exists locally. If the transmitted pose information exists in the positioning server, the electronic device retrieves the transmitted pose information corresponding to the keyframe in the sliding window from the positioning server. If the transmitted pose information does not exist in the positioning server, the positioning server returns a message to the electronic device indicating that no transmitted pose information exists.

[0078] Specifically, before each sliding window optimization, the electronic device queries the positioning server M_LOC to see if there is any transmitted pose information corresponding to the key frame. If the positioning server's M_LOC does not have the transmitted pose information corresponding to the key frame, the electronic device continues to execute the original local sliding window optimization process. If the positioning server's M_LOC has the transmitted pose information corresponding to the key frame, the electronic device obtains the transmitted pose information from the positioning server and records it as T. mloc_wj .

[0079] S104. Determine prior constraints based on the issued pose information and the pose to be optimized.

[0080] It should be noted that during the process of obtaining the transmitted pose information, it is necessary to ensure that the time coordinates of the transmitted pose information and the local pose information are consistent. If the positioning server M_LOC contains transmitted pose information, the electronic device obtains this transmitted pose information from the positioning server. This transmitted pose information is obtained from the local pose information sent by the electronic device to the positioning server M_LOC. For example, if the time node is i and the local pose information is tagged as T... local_wi The lower-level pose information stored after the location service processes its M_LOC is tagged as T. mloc_wi Send pose information T mloc_wiIt includes keyframe information and position information at time node i. The calculation of prior constraints requires sending pose information, uploading pose information locally, and the latest keyframe information in the sliding window. The calculated prior constraints are then added to the sliding window optimization to achieve localization fusion.

[0081] In one embodiment of this disclosure, determining prior constraints based on the transmitted pose information and the pose to be optimized includes: determining prior constraints based on the transmitted pose information, the uploaded pose information, and the pose to be optimized.

[0082] Specifically, prior constraints can be understood as prior probabilities of information based on knowledge of camera frames. For example, an image can usually be represented by a matrix, where each position can take values ​​from 0 to 255. However, a randomly generated matrix is ​​not a natural image; it needs to satisfy many constraints, such as local smoothing. These constraints are derived from people's experience with natural images, i.e., priors. These prior conditions can serve as prior constraints for the actual image. In the embodiments of this disclosure, the prior constraints integrate the transmitted pose information, local pose information, and pose information to be optimized, and together they formulate the prior constraints.

[0083] In one embodiment of this disclosure, the transmitted pose information includes a transmitted pose matrix, the local pose information includes a local pose matrix, and the pose information to be optimized includes a pose matrix to be optimized; determining prior constraints based on the transmitted pose information, the local pose information, and the pose to be optimized includes: determining the product of the transmitted pose matrix, the local pose matrix, and the current pose matrix as prior constraints.

[0084] In this embodiment of the disclosure, the prior constraints for obtaining the issued pose are determined. And add it to the pose information T to be optimized local_wj Above. In the sliding window optimization, As T mloc_wj Prior constraints are incorporated into the pose optimization of the sliding window to achieve the fusion effect. The transmitted pose is denoted as T. mloc_wi, The suffix 'i' is obtained by sending the 'T'. local_wi Obtained.

[0085]

[0086] in,, As described in this disclosure, the matrix calculation method is to issue the pose matrix T. mloc_wi The invertible matrix T of the local pose matrix -1 query_wtmp and the pose matrix T to be optimized local_wj The product of.

[0087] Electronic devices store local pose matrices, so when calculating prior constraints, incorporating the local pose matrix avoids latency issues. Furthermore, the computational cost is minimal compared to the size of the Jacobian matrix required for optimization throughout the sliding window, thus not affecting the overall optimization speed but significantly improving the localization performance.

[0088] S105. Based on the prior constraints, perform sliding window optimization on the pose information to be optimized to obtain the target pose information corresponding to the camera frame.

[0089] The pose determination method provided in this disclosure solves the problem of the timeliness of information transmission being affected by the small sliding window range because the local pose matrix sent from the electronic device to the positioning server M_LOC is saved. Furthermore, this disclosure includes the local pose information corresponding to the previous keyframe in the prior constraints of the sliding window optimization, thus solving the problem of transmission delay. In addition, this disclosure adds the invertible matrix of the local pose matrix corresponding to the keyframe to the prior constraints of the sliding window optimization. Compared to the Jacobian matrix size of the sliding window optimization, the increased computational load in this disclosure will not affect the overall sliding window optimization speed.

[0090] Figure 2 This is a schematic diagram of a pose determination device according to an embodiment of the present disclosure. This embodiment is applicable to pose information processing. The pose determination device can be implemented using software and / or hardware, and the pose information processing device can be configured in an electronic device. The electronic device includes a smart terminal with pose information processing capabilities, including smartphones, laptops, tablets, digital cameras / camcorders, gaming devices, etc. Optionally, the smart terminal includes a touchscreen.

[0091] like Figure 2 The pose determination device 20 provided in this embodiment mainly includes: a camera frame acquisition module 21, a pose determination module 22 to be optimized, a pose acquisition module 23 to be sent, a constraint condition determination module 24, and a target pose generation module 25.

[0092] Among them, the camera frame acquisition module 21 is used to acquire camera frames in the sliding window;

[0093] The pose determination module 22 is used to determine the pose information to be optimized corresponding to the camera frame;

[0094] The pose acquisition module 23 is used to acquire the pose information corresponding to the key frame in the sliding window.

[0095] The constraint determination module 24 is used to determine prior constraints based on the transmitted pose information and the pose to be optimized.

[0096] The target pose determination module 25 is used to perform sliding window optimization on the pose information to be optimized based on the prior constraints to obtain the target pose information corresponding to the camera frame.

[0097] This disclosure provides a pose determination apparatus for performing the following steps: acquiring a camera frame within a sliding window; determining the pose information to be optimized corresponding to the camera frame; acquiring the transmitted pose information corresponding to a keyframe within the sliding window; determining prior constraints based on the transmitted pose information and the pose to be optimized; and performing sliding window optimization on the pose information to be optimized based on the prior constraints to obtain the target pose information corresponding to the camera frame. This disclosure incorporates the transmitted pose information corresponding to the keyframe into the prior constraints, thereby solving the problem of transmission latency.

[0098] In one possible implementation, the apparatus further includes: a keyframe determination module, configured to determine the first frame in the sliding window as a keyframe; and a local pose transmission module, configured to send the local pose information corresponding to the keyframe to the positioning server, so that the positioning server processes the local pose information to obtain the transmitted pose information corresponding to the keyframe.

[0099] Send a pose query request to the positioning server, wherein the pose query request is used to instruct the positioning server to query whether the sent pose information exists on the local end;

[0100] If the transmitted pose information exists in the positioning server, then the transmitted pose information corresponding to the keyframe in the sliding window is obtained from the positioning server.

[0101] The pose acquisition module 23 sends a pose query request to the positioning server, wherein the pose query request is used to instruct the positioning server to query whether the pose information is present on the local end; if the pose information is present on the positioning server, the pose information corresponding to the key frame in the sliding window is obtained from the positioning server.

[0102] In one possible implementation, the positioning server processes the local pose information to obtain the transmitted pose information corresponding to the keyframe, including: if the local pose information is image data, the positioning server processes the image data based on the map data stored locally and the PnP algorithm to obtain the transmitted pose information corresponding to the keyframe.

[0103] In one possible implementation, the positioning server processes the local pose information to obtain the transmission pose information corresponding to the key frame, including: if the local pose information is a pose matrix, the positioning server obtains the timestamp corresponding to the pose matrix; the positioning server obtains the positioning information corresponding to the Global Positioning System based on the timestamp corresponding to the pose matrix, and uses it as the transmission pose information corresponding to the key frame.

[0104] In one possible implementation, the constraint determination module 24 is specifically used to determine prior constraints based on the issued pose information, the local pose information, and the pose to be optimized.

[0105] In one possible implementation, the transmitted pose information includes a transmitted pose matrix, the local pose information includes a local pose matrix, and the pose information to be optimized includes a pose matrix to be optimized; the constraint determination module 24 is specifically used to determine the product of the transmitted pose matrix, the local pose matrix, and the current pose matrix as a priori constraint.

[0106] The pose determination device provided in this embodiment can execute the steps performed in the pose determination method provided in this embodiment, and has the execution steps and beneficial effects, which will not be repeated here.

[0107] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 3 The diagram illustrates a structural schematic suitable for implementing the electronic device 300 in the embodiments of this disclosure. The electronic device 300 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable terminal devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0108] like Figure 3As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303 to implement the image rendering method as described in the embodiments of this disclosure. The RAM 303 also stores various programs and data required for the operation of the terminal device 300. The processing device 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0109] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows terminal device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 A terminal device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0110] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the pose determination method as described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a ROM 302. When the computer program is executed by the processing device 301, it performs the functions defined in the methods of embodiments of this disclosure.

[0111] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0112] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0113] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0114] The aforementioned computer-readable medium carries one or more programs that, when executed by the terminal device, cause the terminal device to: acquire a camera frame in a sliding window; determine the pose information to be optimized corresponding to the camera frame; acquire the downlink pose information corresponding to a keyframe in the sliding window; determine prior constraints based on the downlink pose information and the pose to be optimized; and perform sliding window optimization on the pose information to be optimized based on the prior constraints to obtain the target pose information corresponding to the camera frame.

[0115] Optionally, when one or more of the above-mentioned programs are executed by the terminal device, the terminal device may also execute other steps described in the above embodiments.

[0116] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0118] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0119] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0120] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0121] According to one or more embodiments of this disclosure, this disclosure provides a pose determination method, including: acquiring a camera frame in a sliding window; determining pose information to be optimized corresponding to the camera frame; acquiring downlink pose information corresponding to a keyframe in the sliding window; determining prior constraints based on the downlink pose information and the pose to be optimized; and performing sliding window optimization on the pose information to be optimized based on the prior constraints to obtain target pose information corresponding to the camera frame.

[0122] According to one or more embodiments of this disclosure, a pose determination apparatus is provided. The apparatus includes: a camera frame acquisition module for acquiring a camera frame in a sliding window; a pose determination module for determining pose information to be optimized corresponding to the camera frame; a pose information acquisition module for acquiring pose information to be sent corresponding to a keyframe in the sliding window; a constraint determination module for determining prior constraints based on the sent pose information and the pose to be optimized; and a target pose determination module for performing sliding window optimization on the pose information to be optimized based on the prior constraints to obtain target pose information corresponding to the camera frame.

[0123] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0124] One or more processors;

[0125] Memory, used to store one or more programs;

[0126] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the pose determination methods provided in this disclosure.

[0127] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements a pose determination method as described in any of the present disclosure.

[0128] This disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the pose determination method as described above.

[0129] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0130] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0131] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A pose determination method, said method being applied to visual inertial odometry, characterized in that, include: Capture camera frames within the sliding window; Determine the pose information to be optimized corresponding to the camera frame; Obtain the downlink pose information corresponding to the keyframe in the sliding window, wherein the downlink pose information is the pose information obtained after processing the local pose information corresponding to the keyframe; Based on the issued pose information and the pose to be optimized, determine the prior constraints. Based on the prior constraints, the pose information to be optimized is optimized by sliding window to obtain the target pose information corresponding to the camera frame.

2. The method according to claim 1, characterized in that, The method further includes: The first frame in the sliding window is designated as the keyframe; The local pose information corresponding to the keyframe is sent to the positioning server so that the positioning server can process the local pose information to obtain the sent pose information corresponding to the keyframe.

3. The method according to claim 2, characterized in that, Obtaining the downlink pose information corresponding to the keyframe in the sliding window includes: Send a pose query request to the positioning server, wherein the pose query request is used to instruct the positioning server to query whether the sent pose information exists on the local end; If the transmitted pose information exists in the positioning server, then the transmitted pose information corresponding to the keyframe in the sliding window is obtained from the positioning server.

4. The method according to claim 2, characterized in that, The positioning server processes the local pose information to obtain the transmitted pose information corresponding to the keyframe, including: If the local pose information is image data, the positioning server processes the image data based on the map data stored on its own end and the PnP algorithm to obtain the pose information to be sent to the key frame.

5. The method according to claim 2, characterized in that, The positioning server processes the local pose information to obtain the transmitted pose information corresponding to the keyframe, including: If the local pose information is a pose matrix, the positioning server obtains the timestamp corresponding to the pose matrix; The positioning server obtains the positioning information corresponding to the Global Positioning System based on the timestamp corresponding to the pose matrix, and uses it as the pose information to be sent for the key frame.

6. The method according to claim 2, characterized in that, Based on the transmitted pose information and the pose to be optimized, prior constraints are determined, including: The prior constraints are determined based on the issued pose information, the local pose information, and the pose to be optimized.

7. The method according to claim 6, characterized in that, The transmitted pose information includes a transmitted pose matrix, the local pose information includes a local pose matrix, and the pose information to be optimized includes a pose matrix to be optimized. Based on the transmitted pose information, the local pose information, and the pose to be optimized, prior constraints are determined, including: The product of the issued pose matrix, the local pose matrix, and the pose matrix to be optimized is determined as the prior constraint.

8. A pose determination device, characterized in that, The device is configured in a positioning system including a visual inertial odometry system, comprising: The camera frame acquisition module is used to acquire camera frames that are in the sliding window; The pose determination module is used to determine the pose information to be optimized corresponding to the camera frame; The pose information acquisition module is used to acquire the pose information corresponding to the key frame in the sliding window, wherein the pose information is the pose information obtained after processing the local pose information corresponding to the key frame. The constraint determination module is used to determine prior constraints based on the transmitted pose information and the pose to be optimized. The target pose determination module is used to perform sliding window optimization on the pose information to be optimized based on the prior constraints to obtain the target pose information corresponding to the camera frame.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

11. A computer program product comprising a computer program or instructions that, when executed by a processor, implement the method as described in any one of claims 1-7.

Images