Method and apparatus for estimating the pose of an image acquisition device

Abstract

A method for estimating the posture of an image acquisition device, comprising: acquiring a series of images of multiple landmarks from at least one posture (301); for the most recently acquired N images, determining N matching images by matching each of the N images with a drawn image set, wherein N is an integer greater than 2 (303); and outputting the posture of the image acquisition device by aligning the most recently acquired N images with known posture information of the N matching images (305).

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 976,537, filed on February 14, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure, in some embodiments thereof, relates to computer vision, and more particularly, but not limited to, to methods and apparatus for estimating the pose of an image acquisition device. Background Art

[0004] When implementing virtual reality (VR) and augmented reality (AR), the most critical point is to determine the spatial positioning of the mobile system in real time, for example, estimating the pose of the image acquisition device (such as a camera) provided by the mobile system, where the pose can include its position and rotation. This positioning can be called simultaneous localization and mapping (SLAM).

[0005] In the process of camera and inertial measurement unit (IMU) based localization, the accumulated error may increase over time. In order to suppress the accumulated error, when the camera revisits the same area, the current image pose should be recalibrated according to landmarks (also called feature points). Summary of the invention

[0006] The present disclosure provides a method and apparatus for estimating the position and posture of an image acquisition device.

[0007] In a first aspect, a method for estimating the posture of an image acquisition device is provided, comprising: acquiring a series of images of multiple landmarks from at least one posture; for the most recently acquired N images, determining N matching images by matching each of the N images with a drawn image set, where N is an integer greater than 2; and outputting the posture of the image acquisition device by aligning the most recently acquired N images with known posture information of the N matching images.

[0008] In a second aspect, a device for estimating the posture of an image acquisition device is provided, comprising: an acquisition module configured to acquire a series of images of multiple landmarks from at least one posture; a determination module configured to determine N matching images for the most recently acquired N images by matching each of the N images with a drawn image set, where N is an integer greater than 2; and an output module configured to output the posture of the image acquisition device by aligning the most recently acquired N images with known posture information of the N matching images.

[0009] In a third aspect, an image acquisition device is provided, comprising: a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and run the computer program stored in the memory, thereby implementing the method according to the first aspect or any embodiment thereof.

[0010] According to a fourth aspect, a chip is provided, which is configured to implement the method described in the first aspect or any one of its embodiments.

[0011] Specifically, the chip includes a processor configured to call and run a computer program from a memory, so that a device provided with the chip implements the method described in the first aspect or any one of its embodiments.

[0012] In a fifth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to implement the method described in the first aspect or any one of its embodiments.

[0013] In a sixth aspect, a computer program product is provided, comprising computer program instructions for enabling a computer to implement the method according to the first aspect or any of its embodiments.

[0014] In a seventh aspect, a computer program is provided, which, when executed on a computer, enables the computer to implement the method according to the first aspect or any one of its embodiments.

[0015] Based on the method provided by the embodiment of the present disclosure, it is assumed that the most recently acquired N images are acquired by N cameras mounted on a compact rigid body system, from which 6DoF and scale are estimated. Therefore, even if the pose error accumulates greatly, the accumulated error of the tightly coupled N images is small relative to the entire accumulated pose. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 and Figure 2 The conventional alignment between the current 3D points and the landmarks is exemplarily illustrated.

[0017] Figure 3A sequence diagram of an optional operational flow for estimating the pose of an image acquisition device according to some embodiments of the present disclosure is shown.

[0018] Figure 4 A sequence diagram of an optional operational flow 400 for constructing a rendering image set according to some embodiments of the present disclosure is shown.

[0019] Figure 5 A sequence diagram of an optional operational flow 500 for matching each image to a set of rendered images according to some embodiments of the present disclosure is shown.

[0020] Figure 6 A sequence diagram of an optional operational flow 600 for repositioning an image acquisition device according to some embodiments of the present disclosure is shown.

[0021] Figure 7 An exemplary application scenario for implementing operation 300 according to some embodiments of the present disclosure is shown.

[0022] Figure 8 It is a block diagram of an apparatus for estimating the posture of an image acquisition device according to some embodiments of the present application.

[0023] Fig. 9 is a block diagram of an image acquisition device according to some embodiments of the present application.

[0024] Fig.10 is a block diagram of a chip according to some embodiments of the present application. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure will now be described more fully with reference to the accompanying drawings, in which exemplary embodiments are shown. However, exemplary embodiments of the present disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided to make the present disclosure thorough and complete and to fully convey the concepts of the exemplary embodiments to those skilled in the art. In the accompanying drawings, individual layers and regions are exaggerated for clarity. The same reference numerals in the drawings represent the same elements, and therefore their descriptions will be omitted.

[0026] The described features, structures and / or characteristics of the present disclosure may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are disclosed to provide a thorough understanding of the embodiments of the present disclosure. However, those skilled in the relevant art will recognize that the present disclosure may be implemented without one or more specific details or may be implemented using other methods, components, etc. In other cases, well-known structures, materials or operations are not shown or described in detail to avoid covering up various aspects of the present disclosure.

[0027] In the present disclosure, the terms "connection" and the like should be understood in a broad sense, and unless otherwise specified, can be understood as direct connection or indirect connection through an intermediate medium. The specific meanings of the above terms in the present disclosure can be understood by those skilled in the art according to specific circumstances.

[0028] Further, in the specification of the present disclosure, unless otherwise explicitly stated, "multiple", "a plurality" or "several" means at least two, such as two, three, etc. "and / or" describing the association relationship of associated objects indicates that there may be three relationships, such as A and / or B, indicating that there may be three situations where A alone, B alone, and A and B exist at the same time. The symbol " / " generally indicates that the context objects are in an "or" relationship.

[0029] For simplicity, the term "camera" is used herein to refer to an image acquisition device, such as one or more image sensors, stand-alone cameras, integrated cameras, and / or any sensor suitable for visually recording an object.

[0030] There is an image acquisition system including an image acquisition device (e.g., a camera) in which the pose of the camera in a coordinate system needs to be estimated. Examples of coordinate systems include a world coordinate system and a coordinate system calibrated with the camera pose of the camera when acquiring images. The camera pose is a combination of the position and orientation of the camera relative to the coordinate system. For example, the camera pose x can be represented as a pair of values ​​(R, t), where R is a rotation matrix representing the orientation relative to the coordinate system, and t is a translation vector representing the camera position relative to the coordinate system. Other possible orientation representations are dual angle representations and tensors. Examples of such image acquisition systems are medical systems including cameras inserted into the patient's body (e.g., by swallowing the camera), as well as vehicles and robots, navigation applications, and augmented reality systems. When the camera is operating in an unknown environment, if there is no further information or sensors, the estimation of the camera pose may involve a three-dimensional (3D) reconstruction of the scene. This problem is known as "Simultaneous Localization and Mapping" (SLAM) in the computer vision and robotics community.

[0031] A scene has image features, also called landmarks. An image of a scene captured by a camera (sometimes called a camera view) has observed image features that represent the scene landmarks. Typically, camera pose estimation is solved using bundle adjustment (BA) optimization. Bundle adjustment is a common method for recovering (by estimating) camera poses and reconstructing 3D scenes given a series of images and possible measurements from other sensors. The bundle adjustment optimization calculation aims to minimize the reprojection error of all available images and all landmarks identified in the available images. The reprojection error of a landmark in an image refers to the difference between the position of the observed image features of the landmark in the image and the predicted position of the observed image features of the landmark in the image for a certain camera pose estimate.

[0032] In this disclosure, the terms "visited area", "mapped area" and "observed area" all refer to images of the area captured by a camera and may be used interchangeably.

[0033] The following description refers to a moving camera, but is also applicable to stationary cameras. Estimating the motion of a camera is also called ego-motion estimation.

[0034] Maintaining a high level of estimation accuracy over time is challenging in environments lacking a Global Positioning System (GPS). The accuracy of camera pose estimates and 3D structure from acquired images over a period of time typically deteriorates over time due to the accumulation of estimation errors. Estimation drift is the change in the accumulated estimation error over time. Estimation errors occur both when the camera re-observes a previously mapped area and when the camera continuously explores new areas. When the camera continuously observes new areas, bundle adjustment is reduced to a fixed-lag bundle adjustment, which typically results in rapid track drift. Track drift is the change in the estimate of the camera motion. Re-observing an area is called loop closing. Estimation errors typically decrease when the camera re-observes a previously mapped area, but estimation errors are still unavoidable in large-scale environments, even in the case of loop closing.

[0035] In order to suppress the accumulated error of localization, pose graph methods are widely adopted to reduce the computational cost. The concept of pose graph is to align the current camera trajectory with the old camera trajectory when the camera revisits the same area. This pose graph method tries to increase the similarity between trajectories even if the trajectories have completely different patterns. Although 3D landmarks are taken into account, the distance scaling error between 3D landmarks and the current 3D points constructed by a single camera makes the situation worse.

[0036] Iterative closest point (ICP) for more accurate alignment is an algorithm for minimizing the difference between two point clouds. Typically, ICP constructs the normal vector for each point and then aligns them using matching pairs of points. However, there are some obstacles to directly applying the ICP method due to the very high time cost.

[0037] Another method is Perspective-n-Point (PnP). When the camera revisits the same area, PnP reprojects the constructed 3D landmarks on the current camera image to correct the current camera pose. When the distance between the camera and the 3D landmark increases, the camera pose estimation error also increases.

[0038] Because the above methods only focus on the current (latest) frame pose correction, the recalibrated position may suddenly jump to produce a large translation. In addition, these methods do not consider the distance ratio error between the current estimated 3D point and the 3D landmark.

[0039] For example, Figure 1 and Figure 2 The conventional alignment between the current 3D point and the landmark is illustrated as an example. Figure 1 As shown, the current frame pose 101 is corrected based on the perspective images of landmarks #1, #2, and #3. Figure 2 As shown by the origin of the three arrows in , the recalibrated position of the current frame pose jumps to the left and produces a large translation, which basically does not match the predicted pose and camera trajectory. And, as Figure 2 As shown by the diamonds at the ends of the two arrows in , the distance scale error caused by the change in the current frame pose is not considered in its recalibration, resulting in additional recalibration errors.

[0040] In order to overcome the defects of the above methods, some embodiments of the present disclosure propose a pose estimation scheme that tightly couples the current image frame with its previous image frame. Because the distance between the current trajectory and each camera in the tightly coupled volume is considered as an error, the sudden jump translation problem can be suppressed.

[0041] Some embodiments of the present disclosure also propose a pose estimation scheme that takes into account the distance proportional error between the 3D landmark and the current 3D point when minimizing these errors and estimating the 6 degrees of freedom (DoF) of the tightly coupled body. An example of a pose estimation scheme is performed by the following equation.

[0042] Minimization{3D_error_term+body_pose_error_term+3D_project_error_term} (minimize the sum of 3D error term, pose error term and 3D projection error term),

[0043] Wherein, 3D_error_term = 3D land mark-scale * current 3D point (the 3D error term is equal to the difference between the 3D landmark coordinates and the product of the current 3D point coordinates and the scale).

[0044] body_pose_error_term = scale * body pose - current body pose (the pose error term is equal to the product of pose and scale minus the difference between the current pose), and

[0045] 3D_project_error_term=3D landmark projection on image-current imagepoint (3D projection error term is equal to the difference between the projection coordinates of the 3D landmark on the image and the coordinates of the current image point).

[0046] A method and apparatus for estimating the position and posture of an image acquisition device provided by an embodiment of the present disclosure will be described in detail below in conjunction with the accompanying drawings.

[0047] Reference Figure 3 , which shows a sequence diagram of an optional operational process 300 according to some embodiments of the present disclosure. These embodiments include at least one hardware processor and a single camera. In these embodiments, the at least one hardware processor acquires (301) a series of images of multiple landmarks from at least one pose. In these embodiments, the at least one pose is unknown. It may be necessary to calculate a set of estimates for the at least one pose.

[0048] In some embodiments, for the most recently acquired N images, at least one hardware processor determines (303) N matching images by matching each of the N images with the drawn image set, where N is an integer greater than 2. For example, N = 3. In other words, the three most recently acquired images are matched with the drawn image set respectively to determine three corresponding matching images respectively.

[0049] In some embodiments, at least one hardware processor outputs (305) a pose of the image acquisition device by aligning the most recently acquired N images with known pose information of the N matching images.

[0050] Conventional solutions for recovering the camera's 6 degrees of freedom (DoF) and scaling around loop closures use a single-frame-based PnP approach when computing the absolute pose of the pose graph. Single-frame-based PnP produces jittery motion (wrong 6DoF) when the SLAM-based landmark depth accuracy is poor and the overlap area between the current image frame and the reprojected 3D landmark is less than 60%.

[0051] Based on the method provided by the embodiment of the present disclosure, it is assumed that the most recently acquired N images are acquired by N cameras mounted on a compact rigid body system, from which 6DoF and scale are estimated. Therefore, even if the pose error accumulates greatly, the accumulated error of the tightly coupled N images is small relative to the entire accumulated pose.

[0052] In some embodiments, the drawn image set is acquired during an operation for estimating the pose of the image acquisition device. In other words, the image acquisition device continuously acquires images of a plurality of landmarks and extracts feature information for each acquired image to construct a drawn image set, which is used to estimate the pose of the most recently acquired N images in an on-board manner. In some alternative embodiments, the drawn image set is acquired separately before the operation for estimating the pose of the image acquisition device and is therefore used to estimate the pose of the most recently acquired N images in an off-board manner.

[0053] Reference now Figure 4 , which shows a sequence diagram of an optional operational process 400 for constructing a drawing set according to some embodiments of the present disclosure. These embodiments include at least one hardware processor, a single camera, and a visual inertial odometry (VIO) unit. In these embodiments, at least one hardware processor controls the single camera to capture (401) multiple images of multiple landmarks at different positions of an image acquisition device, and controls the VIO unit to extract (403) feature information for each of the multiple images. In some embodiments, the feature information includes 6DoF information for each image and depth information for multiple landmarks in each image.

[0054] In some embodiments, the VIO unit includes a visual odometry unit and an inertial measurement unit (IMU).

[0055] The exemplary algorithm process of the visual odometry unit is as follows. First, a new frame of image is acquired, and ORB (Oriented FAST and Rotated Brief) feature points are extracted from the image, and the feature descriptions corresponding to the feature points are calculated. Then, the feature points of the most recently acquired image are matched with the feature points of the previous image frame. At the same time, the matched feature points are filtered using the RANSAC algorithm. Finally, the rotation and translation matrix between the current image frame and the previous image frame is obtained by minimizing the reprojection error, thereby obtaining the current position and posture of the image acquisition device.

[0056] In some embodiments, the IMU is configured to obtain the acceleration and angular velocity of the image acquisition device through a gyroscope and an accelerometer, and then calculate the current posture of the image acquisition device through an integral operation. For the sake of brevity, the details are omitted here.

[0057] Reference now Figure 5 , which shows a sequence diagram of an optional operational flow 500 for matching each image to a set of drawn images according to some embodiments of the present disclosure. These embodiments include at least one hardware processor and a single camera. In these embodiments, the at least one hardware processor extracts (501) multiple observed image features of multiple landmarks from multiple images acquired by the single camera from at least one pose. In these embodiments, the at least one pose is unknown. A set of estimates needs to be calculated for at least one pose. Optionally, the observed image features can be expressed in a camera coordinate system.

[0058] In some embodiments, at least one hardware processor extracts (503) a plurality of observed image features by applying an image matching algorithm to the image. The image matching algorithm may include a feature ratio detection algorithm that produces ratio information. Examples of image matching algorithms are SIFT and RANSAC. The at least one hardware processor may identify (505) at least one observed common image feature among the extracted plurality of observed image features, the observed common image features being recorded in at least some of the images. Optionally, the image matching algorithm is used to identify the at least one common feature.

[0059] In some embodiments, operation 500 may be implemented using a corner feature matching and tracking method, such as DBoW2, which is a bag-of-words location recognition method. Based on operation 500, multiple corner features are detected and described by BRIEF descriptors, which are used as visual words to query the drawn image set. DBoW2 may return closed loop candidates after temporal and geometric consistency checks. All BRIEF descriptors may be retained for feature retrieval, but the original image may be ignored to reduce memory consumption. When a loop is detected, a connection is established between the most recently acquired N images and the N matching images by retrieving feature correspondences. The correspondence is found by matching the BRIEF descriptors.

[0060] Reference now Figure 6 , which shows a sequence diagram of an optional operational process 600 for repositioning the pose of an image acquisition device according to some embodiments of the present disclosure. These embodiments include at least one hardware processor. In this column of embodiments, the at least one hardware processor calculates (601) the pose (R, T) based on the following equation:

[0061] Pi=R(p i -p ref )+T;

[0062] (P i -L i )(dot)(P j -L j )-|P i -L i ||P j -L j |ray i (dot)ray j =0;

[0063] R*ray i (dot)(P i -L i )-|P i -L i |=0,

[0064] Where R is a rotation matrix representing the orientation of the image acquisition device relative to the world coordinate system; T is a translation vector representing the position of the image acquisition device relative to the world coordinate system; p ref is the reference position in the volume system when the N most recently acquired images are taken as a whole; p i is the pose translation determined for the i-th image in the N matching images; Pi is the position of the i-th image in the most recently acquired N images; L iis the position of the i-th landmark among the multiple landmarks; dot represents the inner product operation; ray i represents the ray from the camera focus to the image pixel corresponding to the feature point on the i-th image.

[0065] In some embodiments, at least one hardware processor processes the pose (r ref , p ref ) output (603) is:

[0066] r ref =R*R ref ;and

[0067] p ref =T,

[0068] Where R ref is the rotation matrix determined for the most recently acquired image.

[0069] Based on the method provided by the embodiment of the present disclosure, it is assumed that the most recently acquired N images are acquired by N cameras mounted on a compact rigid body system, from which 6DoF and scale are estimated. Therefore, even if the pose error accumulates greatly, the accumulated error of the tightly coupled N images is small relative to the entire accumulated pose.

[0070] Reference now Figure 7 , which illustrates an exemplary application scenario for implementing operation 300 according to some embodiments of the present disclosure. In these embodiments, at least one hardware processor acquires (301) a series of images of landmarks L1, L2, L3 from a series of poses 0, 1, 2, ... 12. In these embodiments, pose 12 is unknown.

[0071] In some embodiments, for the three most recently acquired images, at least one hardware processor determines (303) three matching images by matching each of the three images with the drawn image set. In other words, the three most recently acquired images are matched with the drawn image set to determine the corresponding three matching images. Figure 7 As shown, feature correspondences are found between the three images of poses 10, 11, and 12 and the three matching images of poses 1, 3, and 4, respectively.

[0072] In some embodiments, at least one hardware processor outputs (305) a pose of the image acquisition device by aligning three most recently acquired images with known pose information of three matching images.

[0073] Based on the method provided by the embodiment of the present disclosure, it is assumed that the three most recently acquired images are acquired by three cameras mounted on a compact rigid body system, from which 6DoF and scale are estimated. Therefore, even if the pose error accumulates greatly, the accumulated error of the tightly coupled three images is small relative to the entire accumulated pose.

[0074] The above has been combined Figures 3 to 7 The method embodiment of the present application is described in detail. Figure 8 and Fig. 9 The device / equipment embodiments of the present application are described in detail. It should be understood that the device / equipment embodiments correspond to the method embodiments, and similar descriptions can be referred to the method embodiments.

[0075] Figure 8 is a block diagram of an apparatus for estimating the position and posture of an image acquisition device according to some embodiments of the present application. Figure 8 As shown, the device 800 includes an acquisition module 801 , a determination module 803 and an output module 805 .

[0076] The acquisition module 801 is configured to acquire a series of images of a plurality of landmarks from at least one pose;

[0077] The determination module 803 is configured to determine N matching images for the most recently acquired N images by matching each of the N images with the drawn image set, where N is an integer greater than 2; and

[0078] The output module 805 is configured to output the pose of the image acquisition device by aligning the most recently acquired N images with the known pose information of the N matching images.

[0079] Optionally, in some embodiments, the apparatus 800 further comprises a construction module (not shown). The construction module is configured to acquire multiple images of multiple landmarks at different positions of the image acquisition device and extract feature information for each of the multiple images to obtain the drawn image set.

[0080] Optionally, in some embodiments, the construction module is specifically configured to: calculate 6 degrees of freedom (DoF) information and depth information of the multiple landmarks in each image for each image.

[0081] Optionally, in some embodiments, the building module is specifically configured to calculate the 6DoF information and the depth information based on a visual inertial odometry (VIO) algorithm, utilizing an inertial measurement unit (IMU) and camera image corner feature matching and tracking.

[0082] Optionally, in some embodiments, the determination module 803 is specifically configured to: perform matching of corner point features between each of the N images and each corresponding image in the N matching images.

[0083] Optionally, in some embodiments, the output module 805 is specifically configured to calculate the pose (R, T) according to the following equation:

[0084] Pi=R(p i -p ref )+T;

[0085] (P i -L i )(dot)(P j -L j )-|P i -L i ||P j -L j |ray i (dot)ray j =0;

[0086] R*ray i (dot)(P i -L i )-|P i -L i |=0,

[0087] Where R is a rotation matrix representing the orientation of the image acquisition device relative to the world coordinate system; T is a translation vector representing the position of the image acquisition device relative to the world coordinate system; p ref is the reference position in the volume system when the N most recently acquired images are taken as a whole; p i is the pose translation determined for the i-th image in the N matching images; Pi is the position of the i-th image in the most recently acquired N images; L i is the position of the i-th landmark among the multiple landmarks; dot represents the inner product operation; ray i represents the ray from the camera focus to the image pixel corresponding to the feature point on the i-th image; and

[0088] Output pose (r) for the most recently acquired image ref , p ref )for:

[0089] r ref =R*R ref ;and

[0090] p ref =T,

[0091] Where R ref is the rotation matrix determined for the most recently acquired image.

[0092] Based on the apparatus provided by the embodiment of the present disclosure, it is assumed that the most recently acquired N images are acquired by N cameras mounted on a compact rigid body system, from which 6DoF and scale are estimated. Therefore, even if the pose error accumulates greatly, the accumulated error of the tightly coupled N images is small relative to the entire accumulated pose.

[0093] Fig. 9 is a block diagram illustrating an image acquisition device 900 according to some embodiments of the present application. Fig. 9 The image acquisition device 900 shown includes a processor 910, which can call and run a computer program from a memory to implement a method according to an embodiment of the present application.

[0094] Alternatively, if Fig. 9 As shown, the image acquisition device 900 may further include a memory 920. The processor 910 may call and run a computer program in the memory 920 to implement the method according to the embodiment of the present application.

[0095] The memory 920 may be a separate device from the processor 910 , or may be integrated in the processor 910 .

[0096] Alternatively, if Fig. 9 As shown, the image acquisition device 900 may further include a transceiver 930, and the processor 910 may control the transceiver 930 to communicate with other devices. Specifically, it may send information or data to other devices, or receive other information and data sent by the device.

[0097] The transceiver 930 may include a transmitter and a receiver. The transceiver 930 may also include an antenna, and the number of antennas may be one or more.

[0098] Fig.10 is a block diagram illustrating a chip according to some embodiments of the present application. Fig.10 The chip 1000 shown includes a processor 1010, which can call and run a computer program from a memory to implement a method according to an embodiment of the present application.

[0099] Alternatively, if Fig.10 As shown, the chip 1000 may further include a memory 1020. The processor 1010 may call and run a computer program in the memory 1020 to implement the method according to the embodiment of the present application.

[0100] The memory 1020 may be a separate device from the processor 1010 , or may be integrated in the processor 1010 .

[0101] Optionally, the chip 1000 may further include an input interface 1030. The processor 1010 may control the input interface 1030 to communicate with other devices or chips. Specifically, the processor 1010 may obtain information or data sent by other devices or chips.

[0102] Optionally, the chip 1000 may further include an output interface 1040. The processor 1010 may control the output interface 1040 to communicate with other devices or chips. Specifically, the processor 1010 may output information or data to other devices or chips.

[0103] Optionally, the chip can be applied to the image acquisition device provided in the embodiment of the present application, and the chip can implement the corresponding process implemented by the image acquisition device in the method according to the embodiment of the present application. For the sake of brevity, it will not be described in detail here.

[0104] It should be understood that the chip mentioned in some embodiments of the present application may also be referred to as a system-level chip, a system chip, a chip system or a system on a chip.

[0105] It should be understood that the processor in the embodiment of the present disclosure may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method embodiment can be completed by hardware integrated logic circuits in the processor or instructions in software form. The processor mentioned in some embodiments of the present application may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programming logic devices, discrete gates or transistor logic devices, discrete hardware components, which can implement or implement the methods, steps and block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly presented as being executed and completed by a hardware decoding processor, or by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0106] The memory mentioned in some embodiments of the present application may be a volatile memory or a non-volatile memory, or may include a volatile and non-volatile memory. In some embodiments, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (erasable PROM, EPROM), an electrically erasable programmable read-only memory (EPROM, EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM) used as an external cache. By way of exemplary and non-limiting description, various forms of RAM may be used, for example, static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhancedSDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM), and direct bus random access memory (direct Rambus RAM, DR RAM), etc. It should be noted that memory in the systems and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

[0107] It should be understood that the above-mentioned memory is exemplary rather than restrictive. For example, the memory in the embodiments of the present disclosure may also be a static random access memory (static RAM, SRAM), a dynamic random access memory (dynamic RAM, DRAM), a synchronous dynamic random access memory (synchronous DRAM, SDRAM), a double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), an enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), a synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and a direct bus random access memory (Direct Rambus RAM, DR RAM), etc. In other words, the memory in the embodiments of the present disclosure is intended to include but is not limited to these and any other suitable types of memory.

[0108] The embodiment of the present disclosure also provides a computer-readable storage medium for storing a computer program.

[0109] Optionally, in some embodiments of the present application, a computer-readable storage medium may be applied to a network device, and in some embodiments of the present application, a computer program enables a computer to execute a corresponding process implemented by the network device in each method in some embodiments of the present application. For the sake of brevity, it will not be described here.

[0110] Optionally, in some embodiments of the present application, a computer-readable storage medium may be applied to a mobile terminal / terminal device, and in some embodiments, a computer program enables a computer to execute a corresponding process implemented by the mobile terminal / terminal device in each method in some embodiments of the present application. For the sake of brevity, it will not be described here.

[0111] In some embodiments of the present application, a computer program product is also provided, comprising computer program instructions.

[0112] Optionally, in some embodiments of the present application, the computer program product may be applied to a network device, and the computer program instructions enable a computer to execute the corresponding processes implemented by the network device in each method in some embodiments of the present application. For the sake of brevity, no further description is given here.

[0113] Optionally, in some embodiments of the present application, the computer program product may be applied to a mobile terminal / terminal device, and the computer program instructions enable a computer to execute the corresponding processes implemented by the mobile terminal / terminal device in each method in some embodiments of the present application. For the sake of brevity, no further description is given here.

[0114] In some embodiments of the present application, a computer program is also provided.

[0115] Optionally, in some embodiments of the present application, a computer program may be applied to a network device. When the computer program is run on a computer, the computer is caused to execute the corresponding process implemented by the network device in each method in some embodiments of the present application. For the sake of brevity, it will not be described here.

[0116] Optionally, in some embodiments of the present application, the computer program may be applied to a mobile terminal / terminal device. When the computer program is run on a computer, the computer executes the corresponding process implemented by the mobile terminal / terminal device in each method in some embodiments of the present application. For the sake of brevity, it will not be described here.

[0117] A person skilled in the art can recognize that the units and algorithm steps of each example related to the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software depends on the specific application and design limitations of the technical solution. A person skilled in the art can use different methods to implement the functions described for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] Those skilled in the art can clearly understand that, for the convenience and brevity of explanation, the specific working processes of the above-mentioned systems, devices and units can refer to the corresponding processes in the above-mentioned method embodiments, and will not be repeated here.

[0119] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only exemplary. For example, the division of units is only a division of logical functions, and there may be other divisions in actual implementation. For example, multiple units or components can be combined or integrated into another system, or certain functions can be ignored or not performed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0120] The units described as independent components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one location or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions in the embodiments.

[0121] In addition, the functional units in various embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0122] If these functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in computer-readable storage media. Based on this understanding, an important part of the technical solution of the present application, that is, the part that contributes to the prior art, or other parts of the technical solution, can be embodied in the form of a software product. Computer software products are stored in storage media and include several instructions for enabling a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in some embodiments of the present application. The aforementioned storage media include various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, disks, and optical disks.

[0123] The above contents are only specific implementations of the embodiments of the present application and are not intended to limit the protection scope of the embodiments of the present application. Any modification or replacement made by those skilled in the art within the technical scope disclosed in some embodiments of the present application shall be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for estimating the position and posture of an image acquisition device, comprising: acquiring a series of images of a plurality of landmarks from at least one pose; For the most recently acquired N images, determine N matching images by matching each of the N images with the drawn image set, where N is an integer greater than 2; as well as Outputting the pose of the image acquisition device by aligning the most recently acquired N images with the known pose information of the N matching images, Wherein, aligning the N most recently acquired images with the known pose information of the N matching images to output the pose of the image acquisition device comprises: The pose (R, T) is calculated according to the following equation: Pi=R(p i -p ref )+T; (P i -L i )(dot)(P j -L j )-|P i -L i ||P j -L j |ray i (dot)ray j =0; R*ray i (dot)(P i -L i )-|P i -L i |=0, Where R is a rotation matrix representing the orientation of the image acquisition device relative to the world coordinate system; T is a translation vector representing the position of the image acquisition device relative to the world coordinate system; p ref is the reference position in the volume system when the N most recently acquired images are taken as a whole; p i is the pose translation determined for the i-th image in the N matching images; Pi is the position of the i-th image in the most recently acquired N images; L i is the position of the i-th landmark among the multiple landmarks; dot represents the inner product operation; ray i represents the ray from the camera focus to the image pixel corresponding to the feature point on the i-th image; and Output pose (r) for the most recently acquired image ref , p ref )for: r ref =R*R ref ;and p ref =T, Where R ref is the rotation matrix determined for the most recently acquired image.

2. The method according to claim 1 further comprises obtaining the drawn image set by: capturing multiple images of multiple landmarks at different positions of the image capture device; and Feature information is extracted for each of the plurality of images.

3. The method according to claim 2, wherein: The extracting feature information for each of the multiple images comprises: Six degrees of freedom information and depth information of the plurality of landmarks in each image are calculated for each image.

4. The method according to claim 3, wherein: The calculation steps include: Based on the visual inertial odometry (VIO) algorithm, the 6-DOF information and the depth information are calculated by utilizing the inertial measurement unit (IMU) and camera image corner feature matching and tracking.

5. The method according to any one of claims 1 to 4, wherein: The matching each of the N images with the drawn image set comprises: Matching of corner features is performed between each of the N images and each corresponding image in the N matching images.

6. A device for estimating the position and posture of an image acquisition device, comprising: an acquisition module configured to acquire a series of images of a plurality of landmarks from at least one pose; A determination module configured to determine N matching images by matching each of the N images with the drawn image set for the most recently acquired N images, where N is an integer greater than 2; as well as an output module configured to output the pose of the image acquisition device by aligning the N most recently acquired images with the known pose information of the N matching images, Wherein, the output module is specifically configured as follows: The pose (R, T) is calculated according to the following equation: Pi=R(p i -p ref )+T; (P i -L i )(dot)(P j -L j )-|P i -L i ||P j -L j |ray i (dot)ray j =0; R*ray i (dot)(P i -L i )-|P i -L i |=0, Where R is a rotation matrix representing the orientation of the image acquisition device relative to the world coordinate system; T is a translation vector representing the position of the image acquisition device relative to the world coordinate system; p ref is the reference position in the volume system when the N most recently acquired images are taken as a whole; p i is the pose translation determined for the i-th image in the N matching images; Pi is the position of the i-th image in the most recently acquired N images; L i is the position of the i-th landmark among the multiple landmarks; dot represents the inner product operation; ray i represents the ray from the camera focus to the image pixel corresponding to the feature point on the i-th image; as well as Output pose (r) for the most recently acquired image ref , p ref )for: r ref =R*R ref ;and p ref =T, Where R ref is the rotation matrix determined for the most recently acquired image.

7. The apparatus according to claim 6, further comprising a construction module configured to obtain the drawn image set by: capturing multiple images of multiple landmarks at different positions of the image capture device; and Feature information is extracted for each of the plurality of images.

8. The device according to claim 7, wherein: The building blocks are specifically configured as follows: Six degrees of freedom information and depth information of the plurality of landmarks in each image are calculated for each image.

9. The device according to claim 8, wherein: The building blocks are specifically configured as follows: Based on the visual inertial odometry (VIO) algorithm, the 6-DOF information and the depth information are calculated by utilizing the inertial measurement unit (IMU) and camera image corner feature matching and tracking.

10. The device according to any one of claims 6 to 9, wherein: The determination module is specifically configured as follows: Matching of corner features is performed between each of the N images and each corresponding image in the N matching images.

11. An image acquisition device, comprising a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and run the computer program stored in the memory, thereby implementing the method according to any one of claims 1 to 5.

12. A chip, comprising a processor, wherein the processor is configured to call and run a computer program in a memory, so that a device provided with the chip implements the method according to any one of claims 1 to 5.

13. A computer-readable storage medium for storing a computer program, wherein: The computer program causes a computer to implement the method according to any one of claims 1 to 5.

14. A computer program product comprising computer program instructions for causing a computer to implement the method according to any one of claims 1 to 5.

Images