Planar motion constraint method, electronic device and storage medium

Abstract

The present application relates to a planar motion constraint method, an electronic device and a storage medium, which are applied to a robot system, and the current posture of the camera is obtained by fusion of data of a camera and an inertial measurement sensor. The current posture of the camera is constrained by a plane constraint equation to obtain the motion loss function of the camera. The current posture of the camera, the motion function of the camera and the data of the inertial measurement unit are fused to obtain a fused positioning result of the camera inertial measurement unit with plane constraints, so that the robot has high-precision positioning.

Description

Technical Field

[0001] The present application relates to the field of robot positioning technology, and in particular to a planar motion constraint method, electronic equipment and storage medium. Background Art

[0002] In the related art, cameras and inertial measurement sensors IMU are used as positioning sensors and installed on robots that move in a plane, such as wheeled robots. When using the camera and inertial measurement sensor IMU for fusion positioning, due to the existence of sensor errors and the planar motion restrictions (three degrees of freedom) during the movement of the wheeled robot, there are insufficient constraints in the direction of gravity, and the positioning will drift in the direction of gravity, which will eventually seriously affect the positioning accuracy of the robot. The present invention proposes a planar motion constraint, so that the positioning algorithm can obtain high-precision planar positioning information on the robot that moves in a plane.

[0003] For wheeled robots and other robots with planar motion, most positioning solutions use a three-degree-of-freedom parameter solution. The advantage of this solution is that it has a small amount of calculation and reduces computing consumption. The disadvantage is that when the robot's environment is not completely flat, it will produce large errors. When there are six-degree-of-freedom movements such as slopes, the three-degree-of-freedom positioning model cannot handle it. On the other hand, the six-degree-of-freedom model can model all movements well, but when the six-degree-of-freedom model is used to model a wheeled robot that basically moves in a planar manner, it will be less observable in most cases. Finally, the robot will drift in the other three degrees of freedom, reducing the robot's positioning accuracy.

[0004] Therefore, it is desired to provide a planar motion constraint method to achieve precise positioning of the robot. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present application provides a planar motion constraint method, an electronic device and a storage medium. The planar motion constraint method uses the data fusion of the camera and the inertial measurement sensor to obtain the planar motion loss function of the camera based on the camera's current posture, thereby obtaining the plane-constrained camera inertial measurement sensor fusion positioning result, so that the robot has higher precision.

[0006] The first aspect of the present application provides a planar motion constraint method, which is applied to a robot system and includes the following steps: A planar motion constraint method, which is applied to a robot positioning system, includes the following steps: Step 1, setting the initial state of the camera, the initial state includes: the position of the camera and the posture of the camera; Step 2, predicting the current state of the camera according to the initial state of the camera, and obtaining the current state estimate and covariance of the camera; Step 3, optimizing the current state estimate and covariance of the camera according to the data of the inertial measurement sensor, and obtaining the state transfer function and the covariance transfer function; Step 4, obtaining the depth information of the feature points of the image according to the state transfer function and the covariance transfer function and the feature point pairs of the image taken by the camera; Step 5, obtaining the posture of the camera in the current state through the depth information of the feature points of the image; Step 6, constraining and limiting the posture of the camera in the current state through the planar motion within the time sliding window, and obtaining the plane motion loss function of the camera; Step 7, observing and updating the plane motion loss function of the camera according to the state transfer function and the covariance transfer function and the posture of the camera in the current state, and obtaining the function of the fusion positioning of the camera and the inertial measurement sensor.

[0007] Preferably, the feature point pairs of the images taken by the camera specifically include: step 41, obtaining a first picture taken by the camera in an initial state and a second picture taken by the camera in a current system state; step 42, obtaining feature point pairs of multiple groups of images by performing image feature extraction and matching on the first picture and the second picture.

[0008] Preferably, the method of extracting and matching image features of the first picture and the second picture to obtain feature point pairs of multiple groups of images specifically includes: step 421, extracting corner points of the first picture and the second picture by using the FAST corner point extraction algorithm to obtain corner point information; step 422, obtaining image features of the pictures by establishing a 4-layer image pyramid for the first picture and the second picture; step 423, tracking the movement of corner points between consecutive frame pictures by using a two-frame differential optical flow estimation algorithm based on the corner point information and the image features of the pictures, to obtain feature point pairs of multiple groups of images.

[0009] Preferably, the initial state of the camera is:

[0010] x0={r0,p0};

[0011] P0=R0 -1 R0 -T ;

[0012] Among them, x0 represents the true value of the system state at the initial moment, P0 is the initial covariance matrix of the system state, R0 is the square root of the initial information matrix of the system state, r0 is the initial posture of the camera, and p0 is the initial position of the camera.

[0013] Preferably, the state transfer function and the covariance transfer function are:

[0014]

[0015]

[0016] Δx={Δr,Δp};

[0017] Where Φ and G are system parameters, P i is the covariance matrix of the system at time i, G is the system noise transfer matrix, Represents the state estimation of the camera at the previous moment i-1, It represents the state estimation of the camera at the current time i, Δx is the relative state transition between time i and time i-1, Δr is the relative rotation of the camera's posture, and Δp is the relative displacement of the camera's position.

[0018] Preferably, outliers are filtered out of data points of feature point pairs of multiple groups of images through a RANSAC algorithm based on a basic matrix to obtain a basic matrix; wherein the function of the basic matrix and the feature point pairs of multiple groups of images is: p'Fp=0, wherein p' and p are a pair of feature point pairs, and F is the basic matrix.

[0019] Preferably, step 4, obtaining the depth information of the feature points of the image according to the state transfer function and the covariance transfer function and the feature point pairs of the image captured by the camera, specifically comprises: recovering the depth information through the image frame of the sliding window, and the function of obtaining the depth information is:

[0020]

[0021] Where d is the depth of the feature point to be solved, Ri and ti are the rotation attitude and displacement of the camera respectively, π is the camera projection function, -1 is the reverse camera projection function.

[0022] Preferably, step 5, obtaining the posture of the camera in the current state through the depth information of the feature points of the image specifically includes: restoring the posture information of the camera according to the feature points of the depth information;

[0023] Among them, the camera's posture information is:

[0024]

[0025] Where d is the depth information of the feature point, Ri and ti are the rotation attitude and displacement of the camera respectively, π is the camera projection function, -1 is the reverse camera projection function.

[0026] Preferably, step 6, constraining and limiting the posture of the camera in the current state by plane motion in the time sliding window, and obtaining the plane motion loss function of the camera specifically includes: obtaining the initial plane equation of the camera by plane fitting; obtaining the plane motion loss function of the camera according to the initial plane equation of the camera; wherein the initial plane equation of the camera is:

[0027] (n t , d) = argmin(∑ i n t *p i +d);

[0028] The camera's planar motion loss function is:

[0029] C plane =∑ i r(roll, pitch)*n t *P i +dz+d.

[0030] Preferably, the function of camera and inertial measurement sensor fusion positioning is:

[0031]

[0032]

[0033] A second aspect of the present application provides an electronic device, comprising: a memory; one or more processors; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0034] A third aspect of the present application provides a storage medium storing a computer program, wherein when the computer program is executed by a processor, any of the above-mentioned planar motion constraint methods is implemented.

[0035] The present application provides a planar motion constraint method, an electronic device and a storage medium. The planar motion constraint method uses the current posture of a camera obtained by fusion of data of a camera and an inertial measurement sensor, constrains the current posture of the camera through a plane constraint equation, obtains the motion loss function of the camera, and fuses the current posture of the camera, the motion function of the camera and the data of an inertial measurement unit to obtain a fused positioning result of a camera inertial measurement unit with plane constraints, so that the robot has high-precision positioning.

[0036] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0038] Figure 1 It is a schematic diagram of a flow chart of a planar motion constraint method shown in an embodiment of the present application;

[0039] Figure 2 The embodiment of the present application shows a method without using a planar motion constraint to obtain the x, y, z coordinate trajectory diagram of the camera;

[0040] Figure 3 The embodiment of the present application shows the use of a planar motion constraint method to obtain the x, y, z coordinate trajectory diagram of the camera;

[0041] Figure 4 is a comparison diagram of the z coordinates of the cameras shown in the embodiment of the present application;

[0042] Figure 5 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0044] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0045] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0046] The technical solution of the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0047] Figure 1 1 is a flow chart of a planar motion constraint method shown in an embodiment of the present application. A planar motion constraint method is provided, which is applied to a robot system and includes the following steps:

[0048] Step S1, setting the initial state of the camera, the initial state includes: the position and posture of the camera;

[0049] Specifically, a camera is installed on the robot, and the camera is used to take pictures of the surrounding environment, so as to use the camera to locate the robot's movement.

[0050] In this embodiment, the initial state of the camera is:

[0051] x0={r0,p0};

[0052] P0=R0 -1 R0 -T ;

[0053] Among them, x0 represents the true value of the system state at the initial moment, P0 is the initial covariance matrix of the system state, R0 is the square root of the initial information matrix of the system state, r0 is the initial posture of the camera, and p0 is the initial position of the camera.

[0054] Step S2, predicting the current state of the camera according to the initial state of the camera, and obtaining the current state estimation and covariance of the camera;

[0055] Specifically, in this embodiment, an inertial measurement sensor is used to measure the position of the robot and the speed of the elevator. The inertial measurement sensor includes an accelerometer and a gyroscope. Common index values ​​of the accelerometer include: range, resolution, zero bias, scale factor, cross-coupling coefficient, nonlinearity of scale factor, zero bias stability, zero bias repeatability, bandwidth, operating temperature, etc.

[0056] Based on the initial state of the camera and the data from the fused sensors, such as the speed of the robot measured by the inertial measurement sensor, the state of the camera at the next moment can be inferred. That is, the next system state of the camera is predicted based on the initial state of the camera to obtain the state prediction of the camera.

[0057] Step 3, optimizing the current state estimation and covariance of the camera according to the data of the inertial measurement sensor to obtain a state transfer function and a covariance transfer function;

[0058] Specifically, the state transfer function is:

[0059]

[0060] The camera state transfer refers to calculating the camera state and position at the current moment based on the camera state and position at the previous moment plus the sensor data. Since the camera state and position at each moment are transferred from the previous moment, there will be an initial state, and the subsequent states are transferred step by step from the initial state.

[0061] The camera attitude includes the three-dimensional Euclidean angles of the camera relative to the known object space coordinates, namely the pitch angle, yaw angle, and roll angle.

[0062] The covariance transfer function is:

[0063]

[0064] Δx={Δr,Δp};

[0065] Where Φ and G are system parameters, P i is the covariance matrix of the system at time i, G is the system noise transfer matrix, Represents the state estimation of the camera at the previous moment i-1, It represents the state estimation of the camera at the current time i, Δx is the relative state transition between time i and time i-1, Δr is the relative rotation of the camera's posture, and Δp is the relative displacement of the camera's position.

[0066] Step S4, obtaining depth information of feature points of the image according to the state transfer function and the covariance transfer function and feature point pairs of the image captured by the camera;

[0067] Specifically, the feature point pairs of the image captured by the camera include:

[0068] Step 41, obtaining a first picture taken by the camera in an initial state and a second picture taken by the camera in a current system state;

[0069] Step 42, extracting and matching image features of the first image and the second image to obtain feature point pairs of multiple groups of images.

[0070] In one embodiment, by performing image feature extraction and matching on the first image and the second image, the feature point pairs of multiple groups of images are obtained, which specifically include:

[0071] Step 421, extracting corner points of the first image and the second image by using the FAST corner point extraction algorithm to obtain corner point information;

[0072] Step 422, obtaining image features of the images by establishing a 4-layer image pyramid for the first image and the second image;

[0073] Step 423, based on the corner point information and the image features of the picture, the movement of the corner points between consecutive frame pictures is tracked by a two-frame differential optical flow estimation algorithm to obtain multiple groups of feature point pairs of the images.

[0074] Specifically, for each image, the FAST corner point extraction algorithm is used to extract the corner points in the image. Since the FAST corner point extraction algorithm still extracts some feature points that are not corner points, the Harris corner point response is calculated for the corner points extracted by the FAST corner point extraction, and the feature points that are not corner points are filtered out. In this embodiment, no more than 500 corner points are extracted for each image.

[0075] For the extracted corner points, the two-frame differential optical flow estimation algorithm (LK optical flow estimation algorithm) is used to calculate the movement of the corner points between consecutive frame images. Since the LK optical flow estimation algorithm cannot track the corner points well when they move too fast, an image pyramid is established to stably track the fast-moving corner points. In this embodiment, a 4-layer image pyramid is established for each image.

[0076] In one embodiment, outlier points are filtered out of data points of feature point pairs of multiple groups of images by using a RANSAC algorithm based on a basic matrix to obtain a basic matrix;

[0077] Among them, the function of the basic matrix and the feature point pair of multiple groups of images is:

[0078] p'Fp=0, where p' and p are a pair of feature points, and F is the basic matrix.

[0079] In the present embodiment, the Random Abstract Consensus (RANSAC) algorithm is an iterative algorithm for estimating a mathematical model by using observed data points, wherein the data points include valid points (inliers) and outliers (outliers).

[0080] In this embodiment, abnormal feature matching needs to be removed. This embodiment uses the fundamental matrix as the mathematical model of the matching point, randomly samples 8 groups of point pairs, calculates the satisfied fundamental matrix through the 8 groups of point pairs, and then uses the calculated fundamental matrix to determine whether the point meets the constraints of the fundamental matrix. If it meets the constraints, it is a valid point, and if it does not meet the constraints, it is an abnormal point. Random sampling is performed multiple times, and the fundamental matrix with the most valid points is.

[0081] Step S5, obtaining the posture of the camera in the current state through the depth information of the feature points of the image;

[0082] Specifically, a single-frame image lacks depth information, but multiple-frame images can obtain depth information of the image through triangulation. In this embodiment, through the feature point pairs of two different images, the camera posture is obtained through steps 2 and 3, and the feature points can be triangulated to obtain the depth information of the feature points.

[0083] Specifically, in step 4, according to the state transfer function and the covariance transfer function and the feature point pairs of the image captured by the camera, the depth information of the feature points of the image is obtained, which specifically includes:

[0084] The depth information is restored by sliding the image frame of the window, and the function of obtaining the depth information is:

[0085]

[0086] Where d is the depth of the feature point to be solved, Ri and ti are the rotation attitude and displacement of the camera respectively, π is the camera projection function, -1 is the reverse camera projection function.

[0087] Specifically, due to errors in feature extraction and matching, the camera posture also has errors, which makes the triangulation based on the two images have large errors. In order to obtain more accurate depth information, this embodiment uses a sliding window of image frames for depth recovery, sets the length of the sliding window to 10 frames, and obtains the depth information by minimizing the following equation.

[0088] In one embodiment, obtaining the posture of the camera in the current state through the depth information of the feature points of the image specifically includes:

[0089] Restore the camera's posture information based on the feature points of the depth information;

[0090] Among them, the camera's posture information is:

[0091]

[0092] Where d is the depth information of the feature point, Ri and ti are the rotation attitude and displacement of the camera respectively, π is the camera projection function, -1 is the reverse camera projection function. For the nonlinear equation of the camera's attitude information, the LM method is used to iteratively solve it.

[0093] Step S6, constraining and limiting the posture of the camera in the current state through the plane motion in the time sliding window, and obtaining the plane motion loss function of the camera;

[0094] Specifically, by restoring the camera's posture, the camera's posture has 6 degrees of freedom. However, the robot's planar motion has only 3 degrees of freedom, including the displacement in the z direction, the pitch angle roll, and the pitch angle pitch. These two angles are approximately constant. If these three directions are not constrained, the estimated posture will have a large cumulative error. Therefore, the planar motion within a time sliding window is constrained and limited.

[0095] Specifically, step S6, constraining and limiting the posture of the camera in the current state through the plane motion in the time sliding window, and obtaining the plane motion loss function of the camera specifically includes:

[0096] Through plane fitting, the initial plane equation of the camera is obtained;

[0097] According to the initial plane equation of the camera, the plane motion loss function of the camera is obtained;

[0098] Among them, the initial plane equation of the camera is:

[0099] (n t ,d)=argmin(∑ i n t *p i +d);

[0100] The camera's planar motion loss function is:

[0101] C plane =∑ i r(roll,pitch)*n t *P i +dz+d.

[0102] Among them C plane It represents a constraint that all camera coordinates need to be formed on one plane.

[0103] Step S7, observing and updating the camera's planar motion loss function according to the state transfer function and the covariance transfer function and the camera's posture in the current state, to obtain a function for fusion positioning of the camera and the inertial measurement sensor.

[0104] Specifically, the function of camera and inertial measurement sensor fusion positioning is:

[0105]

[0106]

[0107] in, Represents the constraints of the camera's observation on the camera coordinates.

[0108] Figure 2 This is a method without using a planar motion constraint, and the x, y, z coordinate trajectory diagram of the camera is obtained. Figure 3 The embodiment of the present application shows the use of a planar motion constraint method to obtain the x, y, z coordinate trajectory diagram of the camera. Figure 4 This is a comparison chart of the camera's z coordinate. It can be seen that after using the planar motion constraint method, the drift of the camera's z coordinate is significantly reduced.

[0109] The present application provides a planar motion constraint method, an electronic device and a storage medium. The planar motion constraint method uses the camera attitude variance estimation obtained by the data fusion of the camera and the inertial measurement sensor, and fuses the data of the camera and the inertial measurement sensor IMU to calculate the current positioning coordinates and attitude of the robot. In the traditional visual inertial navigation system using Kalman filtering, a time sliding window is used to estimate the robot attitude in the window in a plane, and the robot's six-degree-of-freedom motion is limited to three degrees of freedom within a certain time range. At the same time, the three-degree-of-freedom planar motion is mathematically modeled as a random process that changes over time. When the robot is in a changing planar motion, such as entering a slope, accurate positioning results can still be obtained, and the positioning drift of the robot in the other three degrees of freedom can be suppressed.

[0110] Figure 5 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application.

[0111] See also Figure 5 , the electronic device 400 includes a memory 410 and a processor 420.

[0112] The processor 420 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0113] The memory 410 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 410 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 410 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0114] The memory 410 stores executable codes, and when the executable codes are processed by the processor 420 , the processor 420 can execute part or all of the methods described above.

[0115] The scheme of the present application has been described in detail above with reference to the accompanying drawings. In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, refer to the relevant description of other embodiments. Those skilled in the art should also know that the actions and modules involved in the description are not necessarily required for the present application. In addition, it can be understood that the steps in the method of the embodiment of the present application can be adjusted in order, merged and deleted according to actual needs, and the modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0116] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0117] Alternatively, the present application can also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or electronic device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0118] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the application herein may be implemented as electronic hardware, computer software, or combinations of both.

[0119] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, a program segment or a code, and the part of the module, the program segment or the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or the flow chart, and the combination of the square boxes in the block diagram and / or the flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A planar motion constraint method, applied to a robot positioning system, characterized in that: The following steps are involved: Step 1, setting the initial state of the camera, the initial state includes: the position and posture of the camera; Step 2, predicting the current state of the camera based on the initial state of the camera, and obtaining the current state estimation and covariance of the camera; Step 3, optimizing the current state estimation and covariance of the camera according to the data of the inertial measurement sensor to obtain a state transfer function and a covariance transfer function; Step 4, obtaining depth information of feature points of the image according to the state transfer function and the covariance transfer function and the feature point pairs of the image taken by the camera; Step 5, obtaining the current posture of the camera through the depth information of the feature points of the image; Step 6, constraining and limiting the camera's posture in the current state through the plane motion in the time sliding window, and obtaining the camera's plane motion loss function; Step 7: Observe and update the camera's planar motion loss function according to the state transfer function, the covariance transfer function, and the camera's current posture to obtain a function for fusion positioning of the camera and the inertial measurement sensor.

2. The planar motion constraint method according to claim 1, characterized in that: The feature point pairs of the image captured by the camera specifically include: Step 41, obtaining a first picture taken by the camera in an initial state and a second picture taken by the camera in a current system state; Step 42, extracting and matching image features of the first image and the second image to obtain feature point pairs of multiple groups of images.

3. The planar motion constraint method according to claim 2, characterized in that: By performing image feature extraction and matching on the first image and the second image, multiple sets of image feature point pairs are obtained, including: Step 421, extracting corner points of the first image and the second image by using the FAST corner point extraction algorithm to obtain corner point information; Step 422, obtaining image features of the images by establishing a 4-layer image pyramid for the first image and the second image; Step 423, based on the corner point information and the image features of the picture, the movement of the corner points between consecutive frame pictures is tracked by a two-frame differential optical flow estimation algorithm to obtain multiple groups of feature point pairs of the images.

4. The planar motion constraint method according to claim 3, characterized in that: The initial state of the camera is: x0={r0,p0}; P0=R0 -1 R0 -T ; Among them, x0 represents the true value of the system state at the initial moment, P0 is the initial covariance matrix of the system state, R0 is the square root of the initial information matrix of the system state, r0 is the initial posture of the camera, and p0 is the initial position of the camera.

5. The planar motion constraint method according to claim 4, characterized in that: The state transfer function and covariance transfer function are: Δx={Δr,Δp}; Where Φ and G are system parameters, P i is the covariance matrix of the system at time i, G is the system noise transfer matrix, Represents the state estimation of the camera at the previous moment i-1, It represents the state estimation of the camera at the current time i, Δx is the relative state transition between time i and time i-1, Δr is the relative rotation of the camera's posture, and Δp is the relative displacement of the camera's position.

6. The planar motion constraint method according to claim 5, characterized in that: The RANSAC algorithm based on the basic matrix is ​​used to filter outliers on the data points of the feature point pairs of multiple groups of images to obtain the basic matrix; Among them, the function of the basic matrix and the feature point pair of multiple groups of images is: p′Fp=0, where p′ and p are a pair of feature points, and F is the basic matrix.

7. The planar motion constraint method according to claim 6, characterized in that: Step 4, obtaining the depth information of the feature points of the image according to the state transfer function and the covariance transfer function and the feature point pairs of the image captured by the camera specifically includes: The depth information is restored by sliding the image frame of the window, and the function of obtaining the depth information is: Where d is the depth of the feature point to be solved, Ri and ti are the rotation attitude and displacement of the camera respectively, π is the camera projection function, -1 is the reverse camera projection function.

8. The planar motion constraint method according to claim 7, characterized in that: Step 5, obtaining the current posture of the camera through the depth information of the feature points of the image, specifically includes: Restore the camera's posture information based on the feature points of the depth information; Among them, the camera's posture information is: Where d is the depth information of the feature point, Ri and ti are the rotation attitude and displacement of the camera respectively, π is the camera projection function, -1 is the reverse camera projection function.

9. An electronic device, characterized in that: include: Memory; one or more processors; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for executing any one of the methods according to claims 1-8.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the planar motion constraint method according to any one of claims 1 to 8 is implemented.

Images