Visual positioning method, device and electronic device based on inverse depth parameter
By updating the feature point information of the road marking point in the visual positioning method of inverse depth parameters, the calculation complexity and positioning inaccurate problems caused by the increase in image frames in the prior art are solved, and higher visual positioning accuracy and information retention are achieved.
Patent Information
- Application Number
- CN202011522724.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-21
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-12-21
AI Technical Summary
The existing visual positioning technology increases the computational complexity when the image frames increase, resulting in inaccurate positioning, loss of road marking point information, and loss of relevant pose constraints.
In the visual positioning method of inverse depth parameters, the road marking feature point in the current image frame is determined, and whether it is a keyframe, the reference frame relative to the previous keyframe or image frame is set, and the road marking feature point information is updated. Optimize camera positioning.
It improves the accuracy of visual positioning, maximizes the use of road marking point constraints, reduces information loss caused by marginalized processing, and improves the positioning accuracy of the system.
Smart Images

Figure CN112686953B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to a visual positioning method, device, electronic device and computer-readable storage medium based on inverse depth parameters. Background Art
[0002] In the prior art, visual positioning technology can usually be divided into two parts: front-end image visual processing and back-end parameter optimization, to estimate the position of the positioning object itself and build a map. In the back-end optimization process, all variables that need to be estimated are treated as parameters with solutions, and then the error function is constructed to find the value of the parameter that minimizes the error function. For example, in SLAM (Simultaneous Localization and Mapping) technology, when solving the back-end optimization problem, the parameters that need to be estimated are the camera pose {Ci} at all times and the location of the landmark points {Lj}, where i = 1, 2, 3, 4, 5..., i is the location of the camera at the i-th moment, j = 1, 2, 3, 4, 5..., j is the j-th landmark point, and the visual residual of the image frame with a common view relationship is calculated. In the prior art, one approach is to use the inverse of the depth corresponding to the landmark point as a parameter, that is, to calculate the visual residual using the inverse depth parameterization method. That is, i represents the first frame in which a certain landmark point is seen, and j represents the subsequent image frame in which the same landmark point is seen. Then, the position of the landmark point is projected to the jth frame, and then projected onto the visual image through the camera's projection model. In this way, the visual residual of the landmark point seen in different image frames can be calculated for visual positioning.
[0003] However, in the visual positioning system, in order to ensure that the computational complexity of the system does not increase with the increase of image frames collected by the camera, only the most recent image frames will be retained as the optimized camera pose, that is, a sequence of image frames consisting of the most recent frames will be maintained. When the number of image frames in the image frame sequence reaches the set value, the first frame will be marginalized according to different strategies, which will result in the loss of image information of some landmarks, the loss of constraints on related poses, and further lead to inaccurate positioning.
[0004] It can be seen that the visual positioning method in the prior art needs to be improved. Summary of the invention
[0005] The embodiment of the present application provides a visual positioning method based on inverse depth parameters, which helps to improve the accuracy of visual positioning.
[0006] In order to solve the above problems, in a first aspect, an embodiment of the present application provides a visual positioning method based on inverse depth parameters, including:
[0007] In the process of visual positioning based on the inverse depth parameter, in response to obtaining a current image frame captured by the camera, determining a feature point corresponding to a landmark point observed in the current image frame;
[0008] Determine whether the current image frame is a key frame;
[0009] In response to the current image frame being a key frame, setting a reference frame of a feature point corresponding to a newly added landmark point observed in the current image frame relative to a previous key frame as the current image frame;
[0010] In response to the current image frame not being a key frame, setting a reference frame of a feature point corresponding to a newly added landmark point observed in the current image frame relative to a previous image frame as the current key frame;
[0011] The information of the feature points corresponding to the landmark points observed in the current image frame is used to update the information of the feature points corresponding to the landmark points observed in the image frame sequence, so that the preset visual positioning engine positions the camera based on the updated information of the feature points corresponding to the landmark points.
[0012] In a second aspect, an embodiment of the present application provides a visual positioning device based on an inverse depth parameter, comprising:
[0013] A current image frame acquisition module is used to determine the feature points corresponding to the landmark points observed in the current image frame in response to acquiring the current image frame captured by the camera during the visual positioning based on the inverse depth parameter;
[0014] A key frame determination module, used to determine whether the current image frame is a key frame;
[0015] A first reference frame setting module, configured to, in response to the current image frame being a key frame, set a reference frame of a feature point corresponding to a newly added landmark point observed in the current image frame relative to a previous key frame as the current image frame;
[0016] A second reference frame setting module, configured to set, in response to the current image frame not being a key frame, a reference frame of a feature point corresponding to a newly added landmark point observed in the current image frame relative to a previous image frame as the current key frame;
[0017] The feature point information updating module is used to update the information of the feature points corresponding to the landmark points observed in the image frame sequence through the information of the feature points corresponding to the landmark points observed in the current image frame, so that the preset visual positioning engine can position the camera based on the updated information of the feature points corresponding to the landmark points.
[0018] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the visual positioning method based on inverse depth parameters described in the embodiment of the present application is implemented.
[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the visual positioning method based on inverse depth parameters disclosed in an embodiment of the present application.
[0020] The visual positioning method based on inverse depth parameters disclosed in the embodiment of the present application determines the corresponding feature points of the landmark points observed in the current image frame in response to obtaining the current image frame captured by the camera during the visual positioning based on the inverse depth parameters; determines whether the current image frame is a key frame; in response to the current image frame being a key frame, sets the reference frame of the newly added landmark points corresponding to the feature points observed in the current image frame relative to the previous key frame as the current image frame; in response to the current image frame not being a key frame, sets the reference frame of the newly added landmark points corresponding to the feature points observed in the current image frame relative to the previous image frame as the current key frame; updates the information of the landmark points corresponding to the feature points observed in the image frame sequence through the information of the landmark points corresponding to the feature points observed in the current image frame, so that the preset visual positioning engine positions the camera based on the updated information of the landmark points corresponding to the feature points, which helps to improve the accuracy of visual positioning.
[0021] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0023] Figure 1 is a flow chart of a visual positioning method based on inverse depth parameters according to Embodiment 1 of the present application;
[0024] Figure 2 is a schematic diagram of image frames captured by a camera in Embodiment 1 of the present application;
[0025] Figure 3 is a schematic diagram of characteristic points of landmark points in different image frames in Example 1 of the present application;
[0026] Figure 4 is a schematic diagram of the structure of a visual positioning device based on inverse depth parameters according to Embodiment 2 of the present application;
[0027] Figure 5 A block diagram schematically shows an electronic device for executing the method according to the present application; and
[0028] Figure 6 A storage unit for holding or carrying a program code for implementing the method according to the present application is schematically shown. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0030] Embodiment 1
[0031] The embodiment of the present application discloses a visual positioning method based on inverse depth parameters, such as Figure 1 As shown, the method includes: steps 110 to 150.
[0032] Step 110, in the process of visual positioning based on inverse depth parameters, in response to obtaining a current image frame captured by a camera, determining a feature point corresponding to a landmark point observed in the current image frame.
[0033] Taking visual positioning as an example, the current visual positioning problems (VISLAM or VIO) can be divided into two parts, namely the visual processing of the front-end image and the back-end optimization method to estimate its own position.
[0034] In the optimization backend, all variables that need to be estimated are treated as parameters with solutions, and then the error function is constructed to find the value of the parameter that minimizes the error function. For example, in the SLAM problem, the parameters that need to be estimated are the camera poses {Ci} at all times, where i = 1, 2, 3, 4, 5..., i is the position of the camera at the i-th moment, and the position of the landmarks {Lj}, j = 1, 2, 3, 4, 5..., where j is the j-th landmark. The entire optimization problem is transformed into the following formula:
[0035]
[0036] in, represents the residual of the prior information generated for the first frame, represents the visual residual, V represents the constraints on the position, velocity and attitude between two adjacent frames generated by the navigation system, that is, the residual generated by the navigation system; i Represents the collection of other image frames that have a co-viewing relationship with the i-th frame. Among them, in the way of visual residual representation, one method is to use the image frame where the landmark point is first seen as the reference frame, and the inverse of its corresponding depth as the parameter, that is, the inverse depth parameter. When the inverse depth parameterization method is used, in the visual residual term, i represents the first image frame that sees a certain landmark point, and j represents the subsequent image frame that sees the same landmark point. Then, by projecting the landmark point position to the j-th frame, and then projecting it to the visual image through the camera's projection model, the visual residual term, also called the reprojection error, can be calculated. This relationship can generally be expressed as follows Figure 2 The form shown.
[0037] like Figure 2 As shown, L1 and L2 identify two landmarks, where landmark L1 is observed by the camera in the image frames captured at times C0, C1, C2, and C3, and landmark L2 is observed by the camera in the image frames captured at times C2, C3, C4, and C5. When the reprojection error is represented by the inverse depth parameter, each residual will be related to the camera position and posture at two moments. For example, for landmark L1, one residual is generated in the image frames captured at C0 and C1, one residual is generated in the image frames captured at C0 and C2, and one residual is generated in the image frames captured at C0 and C3, for a total of three residuals. The visual residuals of the landmarks generated in the image frames captured at different times will be used as parameters to locate the camera.
[0038] In the process of visual positioning based on inverse depth parameters, the preset visual positioning engine positions the camera according to feature point information corresponding to a series of landmark points observed in the image frame sequence containing the landmark points collected by the camera.
[0039] In some embodiments of the present application, the feature point corresponding to the landmark point refers to the image of the landmark point in the image frame, wherein the information of the landmark point based on the feature point of the specified image frame includes: the planar position coordinates and the inverse depth value of the feature point in the specified image frame, wherein the inverse depth value is the inverse of the depth of the landmark point relative to the current camera position.
[0040] For example, for a landmark point p observed in the i-th image frame, its image coordinate point in the i-th image frame is represented by qi, and the image coordinate point qi is first back-projected onto the unit plane (i.e., a plane with a depth of 1) to obtain the image coordinate point qi', i.e., the feature point of the landmark point p. The coordinates of the image coordinate point qi' can be expressed as [x, y, 1] T , then the inverse depth of the landmark point p is λ, where x, y are the image coordinates of the landmark point in the i-th frame image, that is, the plane position coordinates of the image coordinate point qi, and T is the transposed matrix. Then the three-dimensional coordinate position of the image coordinate point qi (that is, the feature point of the landmark point) can be expressed as [x / λ, y / λ, 1 / λ] T .
[0041] According to the above method, an expression representing the coordinates of the landmark point can be established. That is, the spatial position of the landmark point can be determined by the inverse depth λ and the camera pose corresponding to the i-th frame image.
[0042] During the visual positioning process, the camera collects image frames containing landmark points in real time and stores them in the image frame sequence. In addition, according to the above method, information of feature points corresponding to the landmark points in the collected current image frame is determined in real time.
[0043] Step 120: Determine whether the current image frame is a key frame.
[0044] In the process of capturing image frames by the camera, usually, there is a key frame every certain number of image frames. In specific implementation, it can be determined whether the currently captured image frame is a key frame according to the definition of the image capture system or according to the video transmission protocol.
[0045] Step 130 , in response to the current image frame being a key frame, setting a reference frame of feature points corresponding to newly added landmark points observed in the current image frame relative to a previous key frame as the current image frame.
[0046] Normally, in order to ensure that the computational complexity does not increase with the increase in the number of camera positions and to save storage space, an upper length limit will be set for the image frame sequence used for visual positioning while ensuring positioning accuracy. When the image frames in the image frame sequence reach the set upper length limit, the image frames in the image frame sequence will be marginalized in a first-in-first-out manner, that is, the earliest collected image frames will be deleted. In the embodiment of the present application, this upper length limit is defined as the upper limit threshold of marginalization processing. Specifically, for example, the upper limit threshold of marginalization processing can be set to 20, that is, the preset visual positioning engine locates the camera according to the landmark points observed in the latest 20 frames of image frames collected.
[0047] During the visual positioning process, when the camera collects image frames containing landmarks in real time and stores them in the image frame sequence, if the length of the image frame sequence reaches the upper limit threshold of the marginalization processing, the image frame sequence needs to be maintained, for example, marginalization processing is performed to delete the earliest collected image frame.
[0048] Afterwards, the preset visual positioning engine solves the above optimization problem, iteratively solves the parameters to be determined, and obtains the position of the camera at a specified time, thereby completing the positioning of the camera.
[0049] In the prior art, when performing visual positioning, the feature point information of each landmark point observed in the image frame sequence is represented based on the first image frame in which the landmark point is observed. As can be seen from the above description, when the image frame captured at time C0 is marginalized, since the landmark point L1 depends on the image frame captured at time C0, or each visual residual is related to the image frame captured at time C0, the landmark point L1 must be marginalized at the same time to ensure that the optimized residuals can be retained. However, this will result in a situation where a landmark point is tracked very well, but good constraints cannot be provided all the time. That is, when the image frame captured at time C0 is marginalized, the landmark point L1 is tracked very well, but it must be marginalized, thereby losing the opportunity to update the landmark point L1, and also losing the opportunity to constrain other postures related to the landmark point L1.
[0050] In some embodiments of the present application, in order to improve the accuracy of visual positioning, the corresponding feature points of landmark points with good tracking status are retained as much as possible. During the image frame acquisition process, for the corresponding feature points of new landmark points observed in the image frames between two key frames and also observed in the next key frame, the reference frame of the feature point is set to the next key frame, that is, the latest key frame in which the corresponding feature points of the landmark points are observed.
[0051] In some embodiments of the present application, after determining the landmark points observed in the current image frame, in response to the current image frame being a key frame, the reference frame of the feature points corresponding to the newly added landmark points observed in the current image frame relative to the previous key frame is set as the current image frame, including: determining a landmark point that was not observed in the image frame previous to the current image frame and observed in the current image frame as the first newly added landmark point; setting the reference frame of the feature points corresponding to the first newly added landmark point to the current image frame; determining a landmark point that was first observed in an image frame captured between the previous key frame and the current image frame and observed in the current image frame as the second newly added landmark point; and converting the reference frame of the feature points corresponding to the second newly added landmark point to the current image frame.
[0052] The newly added landmark points corresponding to feature points include landmark points corresponding to feature points that are observed for the first time in the current image frame, and also include landmark points corresponding to feature points that are observed for the first time in an image frame captured after the most recent key frame and before the current image frame, and are also observed in the current image frame (i.e., co-viewed).
[0053] For the feature points corresponding to the landmark points observed for the first time in the current image frame, the reference frame of the feature points corresponding to the landmark points is directly set as the current image frame (i.e., the current key frame); for the feature points corresponding to the landmark points observed for the first time in the ordinary image frame, when they are observed, their reference frame has been set as the image frame in which the feature points corresponding to the landmark points are observed, and the reference frame of the feature points corresponding to the landmark points needs to be converted into the current image frame (i.e., the current key frame).
[0054] Take the image frame sequence for visual positioning as an example, which includes 20 frames of image frames. Assume that the current time is t, and the image frame captured by the camera at the current time is a key frame, which is recorded as the current key frame. Assume that the image frame captured by the camera at the tnth time (where n is an integer greater than 2) is a key frame, and the camera has captured a total of n-1 frames of image frames from the t-n+1th to t-1th time. If landmark points L1 and L2 are observed in the key frame captured by the camera at the tnth time, landmark points L1 and L2 are also observed in the image frames captured at the t-n+1th to t-2th time, and landmark points L1 and L2 are observed in the image frames captured at the t-1th time and the current time, as well as landmark point L3, then landmark point L3 is a newly added landmark point. For newly added landmark points, the positioning system will record the first image frame in which the newly added landmark point is observed as the reference frame of the feature point corresponding to the newly added landmark point. Specifically for this embodiment, after acquiring the image frame captured by the camera at time t-1, the newly added landmark point L3 observed in the image frame captured at that time will be recorded, and the reference frame mark of the feature point of landmark point L3 will be set to the image frame captured at time t-1.
[0055] If the currently captured image frame is a key frame, the reference frame of the feature point of the landmark point L3 is converted into the current image frame, that is, the image frame captured at time t, which is also a key frame.
[0056] Assume that the image frame captured by the camera at time tn (where n is an integer greater than 2) is a key frame, and the camera has captured a total of n-1 image frames at time t-n+1 to t-1. If landmark points L1 and L2 are observed in the key frame captured by the camera at time tn, landmark points L1 and L2 are also observed in the image frames captured at time t-n+1 to t-1, and landmark points L1 and L2 are observed in the image frames captured at the current time (i.e., time t), landmark point L3 is a newly added landmark point. For newly added landmark points, the positioning system will record the first image frame in which the newly added landmark point is observed as the reference frame of the feature point corresponding to the newly added landmark point, that is, the image frame captured at the current time (i.e., time t) is set as the reference frame of the feature point of the newly added landmark point L3.
[0057] The landmark points observed in each collected image frame are mapped to the most recently collected key frame. The key frame carries the landmark point information to avoid the loss of landmark point information due to the marginalization of the image frame in which a landmark point is observed during the marginalization process, thereby further losing the positioning constraint corresponding to the landmark point.
[0058] When converting a reference frame of a feature point corresponding to a landmark point into other image frames, it is necessary to simultaneously update the inverse depth value and the variance of the inverse depth value of the feature point corresponding to the landmark point, so that the preset visual positioning engine can position the camera based on the information of the updated feature point corresponding to the landmark point.
[0059] In some embodiments of the present application, the reference frame of the feature point corresponding to the newly added landmark point is set as the current image frame, including: using the reference frame of the feature point corresponding to the newly added landmark point as the first reference frame, using the current image frame as the second reference frame, and updating the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame to an inverse depth value based on the second reference frame.
[0060] by Figure 3 Taking the reference frame switching diagram shown in the figure as an example, P is a landmark point, and its three-dimensional coordinate position can be expressed as [x / λ, y / λ, 1 / λ] T , the left side is the image frame C0 of the landmark point collected, and the right side is the image frame C1 of the landmark point collected. Then, the visual residual e of the landmark point in the two image frames is the parameter term in the above optimization problem. It can be expressed by the projection formula: p2-p2'=p2-K*[x / λ,y / λ,1 / λ] T, where K is the camera's attitude rotation matrix, p1 is the plane position coordinate of the landmark point p observed in the image frame C0, p2 is the plane position coordinate after p1 is mapped to the image frame C1, and p2' is the plane position coordinate of the landmark point p observed in the image frame C1. It can be seen that the visual residual can be represented by the inverse depth value. Among them, the plane position coordinate corresponds to the camera's posture.
[0061] In some embodiments of the present application, updating the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame to the inverse depth value based on the second reference frame includes: converting the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame to the inverse depth value based on the second reference frame through a preset inverse depth conversion function; wherein the preset inverse depth conversion function performs the following operations: determining the spatial position information of the feature point corresponding to the landmark point based on the second reference frame according to the spatial position of the feature point corresponding to the landmark point based on the first reference frame, the camera posture rotation matrix corresponding to the first reference frame, and the visual residual between the first reference frame and the second reference frame of the feature point corresponding to the landmark point; determining the inverse depth value of the feature point corresponding to the landmark point based on the second reference frame according to the spatial position information of the feature point corresponding to the landmark point based on the second reference frame and the camera posture rotation matrix corresponding to the second reference frame.
[0062] In some embodiments of the present application, the preset inverse depth conversion function can be expressed as:
[0063] λ'=1 / [(R1 T (R0*[x / λ,y / λ,1 / λ] T +p0-p1)).z()];
[0064] Where [x / λ, y / λ, 1 / λ] T is the spatial position of the feature point corresponding to the landmark point based on the first reference frame, λ is the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame, R0 is the camera attitude rotation matrix corresponding to the first reference frame, R1 is the camera attitude rotation matrix corresponding to the second reference frame, P0 is the camera pose corresponding to the first reference frame, P1 is the camera pose corresponding to the second reference frame, T is the transposed matrix, and z() is a function of taking the depth value. Among them, p0-p1 is the visual residual between the first reference frame and the second reference frame of the feature point corresponding to the landmark point.
[0065] In some embodiments of the present application, after updating the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame to the inverse depth value based on the second reference frame, it also includes: determining the variance of the inverse depth value of the feature point corresponding to the landmark point based on the second reference frame according to the variance of the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame and the Jacobian matrix of the preset inverse depth conversion function on the inverse depth value; and updating the variance of the feature point corresponding to the landmark point in the preset visual positioning engine parameters according to the variance of the inverse depth value of the feature point corresponding to the landmark point based on the second reference frame.
[0066] From the above positioning problem, we know that the variance of the inverse depth value is also a parameter of the preset visual positioning engine. Therefore, after updating the inverse depth value, the variance of the inverse depth value must also be updated. For example, through the formula v'=J(λ)*v*J(λ) T .z() updates the variance of the inverse depth value, where λ is the inverse depth value of the landmark feature point based on the first reference frame, v is the variance of the inverse depth value in the first reference frame, v' is the variance of the inverse depth value in the first reference frame, J(λ) is the partial derivative of the preset inverse depth conversion function on the inverse depth value (i.e., the Jacobian matrix), T is the transposed matrix, and z() is a function of taking depth values.
[0067] Step 140: In response to the current image frame not being a key frame, a reference frame of feature points corresponding to newly added landmark points observed in the current image frame relative to a previous image frame is set as the current image frame.
[0068] In some embodiments of the present application, after the landmarks observed in the current image frame are determined, in response to the current image frame not being a key frame, the reference frame of the feature points corresponding to the newly added landmarks observed in the current image frame relative to the previous image frame is set as the current image frame, including: determining the landmarks corresponding to the feature points that were not observed in the previous image frame of the current image frame and observed in the current image frame as the newly added landmarks corresponding to the feature points; and setting the reference frame of the newly added landmarks corresponding to the feature points to the current image frame. That is, for the landmarks corresponding to the feature points observed for the first time in the current image frame, the reference frame of the landmarks corresponding to the feature points is directly set to the current image frame.
[0069] In some embodiments of the present application, feature point information corresponding to each image frame of each landmark point observed in each image frame collected during the positioning process can be stored in a feature point tracking list. For example, the correspondence between the identifier of each image frame in which the landmark point is observed (which may include the timestamp of image frame acquisition, and the image frame category identifier of whether it is a key frame) and the landmark point, as well as the landmark point information based on the corresponding image frame (such as the plane position coordinates, inverse depth value), and other information used for positioning (such as the variance of the inverse depth value, etc.). When determining whether a newly added landmark point is observed in a certain image frame, the lost or newly added landmark point can be determined based on the correspondence between the identifiers of adjacent image frames and the landmark points.
[0070] Step 150, updating the information of the feature points corresponding to the landmark points observed in the image frame sequence through the information of the feature points corresponding to the landmark points observed in the current image frame, so that the preset visual positioning engine positions the camera based on the updated information of the feature points corresponding to the landmark points.
[0071] The aforementioned steps can be used to set the reference frame of the feature points corresponding to the newly added landmark points in the acquired current image frame. Furthermore, if the current image frame is a key frame, for some landmark points corresponding to feature points that are observed for the first time in other image frames and are co-visible in the current image frame, their reference frames are converted to the current image frame. That is, the current image frame may carry information on feature points corresponding to landmark points that are in a good tracking state. Next, the current image frame needs to be stored in the image frame sequence, and at the same time, the information on feature points corresponding to the landmark points observed in each image frame in the image frame sequence needs to be updated.
[0072] As mentioned above, in order to ensure that the computational complexity does not increase with the increase in the number of camera positions and to save storage space, an upper length limit is set for the image frame sequence used for visual positioning. When the image frames in the image frame sequence reach the set upper length limit, the image frames in the image frame sequence are marginalized in a first-in-first-out manner, that is, the earliest captured image frame is deleted. Therefore, when inserting the current image frame into the image frame sequence, it is necessary to determine whether the length of the image frame sequence reaches the upper limit threshold of marginalization processing, and when the length of the image frame sequence reaches the upper limit threshold of marginalization processing, the image frame sequence and the landmark feature point information corresponding to each image frame in the image frame sequence are maintained.
[0073] In some embodiments of the present application, information on feature points corresponding to landmark points observed in the image frame sequence is updated through information on feature points corresponding to landmark points observed in the current image frame, including: adding the current image frame to the image frame sequence; determining marginalized image frames in the image frame sequence in response to the length of the image frame sequence reaching an upper threshold of marginalization processing; converting a reference frame of feature points corresponding to landmark points using the marginalized image frame as a reference frame to other image frames in the image frame sequence, wherein the other image frames refer to image frames in the image frame sequence other than the marginalized image frame.
[0074] After setting the reference frame based on the current image frame, finally, the acquired current image frame is added to the image frame sequence used for visual positioning. After that, it is further determined whether the length of the image frame sequence reaches a set length threshold after the current image frame is added. If the set length threshold is reached, the image frame sequence needs to be optimized, such as performing edge processing.
[0075] In some embodiments of the present application, the marginalized image frame is usually the image frame with the earliest acquisition time in the image frame sequence. Therefore, the marginalized image frame can be determined by the timestamp of each image frame in the image frame sequence.
[0076] For the marginalized image frames, since the feature point information of the landmark points observed for the first time in the image frame is used in the positioning process, that is, the feature point information of the landmark points with the image frame as the reference frame is used, in order to retain the constraints of the landmark points on positioning, it is necessary to convert the feature point information of the landmark points with the image frame as the reference frame into other image frames as reference frames, and recalculate the feature point information of the landmark points.
[0077] In specific implementation, according to the type attribute of the marginalized image frame and the tracking state of the feature point corresponding to the landmark point with the marginalized image frame as the reference frame, the information of the feature point corresponding to the landmark point is updated. The following introduces several methods for updating the feature point information.
[0078] In some embodiments of the present application, the conversion of the reference frame of the landmark point corresponding to the marginalized image frame as the reference frame into other image frames in the image frame sequence includes: for each landmark point corresponding to the marginalized image frame as the reference frame, determining whether the other image frames in the image frame sequence include a key frame in which the landmark point is observed; in response to the other image frames including the key frame in which the landmark point is observed, ending the operation of updating the information of the landmark point corresponding to the feature point observed in the image frame sequence.
[0079] Taking the current moment as moment t, the marginalized image frame as the image frame captured at moment tn, and the landmark point L1 observed in the marginalized image frame as an example, that is, the marginalized image frame is the reference frame of the feature point p1 of the landmark point L1, for the landmark feature point p1 with the image frame as the reference frame, first determine the key frame in the image frame captured from moment t-n+1 to moment t that shares the landmark point L1 with the marginalized image frame (for example, determine the key frame in the image frame captured from moment t-n+1 to moment t that serves as the reference frame of the landmark feature point p1). If there is a key frame that shares the landmark point L1 with the marginalized image frame, then as described above, the reference frame update operation performed when the current image frame is obtained as the key frame, the reference frame of the landmark feature point with the marginalized image frame as the reference frame has been converted to the reference frame with the most recent and latest captured key frame as the reference frame. Therefore, in this step, there is no need to perform any operation on the landmark feature point information with the marginalized image frame as the reference frame.
[0080] In some embodiments of the present application, when the current image frame is acquired, the step of determining whether the current image frame is a key frame may not be performed. Accordingly, in the process of marginalization processing, the reference frame of the feature points corresponding to the landmark points with the marginalized image frame as the reference frame is converted into other image frames in the image frame sequence, including: for each feature point corresponding to the landmark point with the marginalized image frame as the reference frame, determining whether the other image frames in the image frame sequence include a key frame in which the landmark point is observed; in response to the other image frames including the key frame in which the landmark point is observed, determining the key frame with the earliest acquisition time among the key frames in which the landmark point is observed as the second reference frame, and using the marginalized image frame as the first reference frame; updating the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame to the inverse depth value based on the second reference frame.
[0081] Still taking the current moment as moment t, the marginalized image frame as the image frame captured at moment tn, and the landmark point L1 observed in the marginalized image frame as an example, that is, the marginalized image frame is the reference frame of the feature point p1 of the landmark point L1, for the landmark feature point p1 with this image frame as the reference frame, first determine the key frame in the image frames captured from moment t-n+1 to moment t that shares the landmark point L1 with the marginalized image frame (for example, determine the key frame in the image frames captured from moment t-n+1 to moment t that serves as the reference frame for the landmark feature point p1), if there is a key frame that shares the landmark point L1 with the marginalized image frame, then the key frame that is closest to the marginalized image frame in terms of acquisition time and shares the landmark point L1 with the marginalized image frame is used as the reference frame for the feature point of the landmark point L1.
[0082] When modifying the reference frame of a feature point corresponding to a landmark point, a reference frame with an earlier acquisition time is used as the first reference frame (such as a marginalized image frame), and a determined key frame is used as the second reference frame. The inverse depth value of the feature point corresponding to the landmark point based on the first reference frame is updated to an inverse depth value based on the second reference frame, thereby updating the inverse depth parameters used by the preset visual positioning engine.
[0083] In some embodiments of the present application, the reference frame of the feature points corresponding to the landmark points with the marginalized image frame as the reference frame is converted into other image frames in the image frame sequence, including: for each feature point corresponding to the landmark point with the marginalized image frame as the reference frame, determining whether the other image frames in the image frame sequence include a key frame in which the landmark point is observed; in response to the other image frames not including the key frame in which the landmark point is observed, determining the other image frame with the earliest acquisition time among the other image frames in which the landmark point is observed as the second reference frame, and using the marginalized image frame as the first reference frame; updating the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame to an inverse depth value based on the second reference frame.
[0084] Take the current time as time t, the marginalized image frame as the image frame collected at time tn, and the landmark point L1 observed in the marginalized image frame as an example, that is, the marginalized image frame is the reference frame of the feature point p1 of the landmark point L1. For the landmark feature point p1 with this image frame as the reference frame, first determine the key frame that shares the landmark point L1 with the marginalized image frame in the image frames collected from time t-n+1 to time t (for example, determine the key frame that serves as the reference frame of the landmark feature point p1 in the image frames collected from time t-n+1 to time t). If there is no key frame that shares the landmark point L1 with the marginalized image frame, further determine the image frame that shares the landmark point L1 with the marginalized image frame in the image frames collected from time t-n+1 to time t (for example, determine the image frame that serves as the reference frame of the landmark feature point p1 in the image frames collected from time t-n+1 to time t). If there is an image frame that shares the landmark point L1 with the marginalized image frame, the image frame that is closest to the marginalized image frame in acquisition time and shares the landmark point L1 with the marginalized image frame is used as the reference frame for the feature point of the landmark point L1.
[0085] When modifying the reference frame of a feature point corresponding to a certain landmark point, a reference frame with an earlier acquisition time is used as the first reference frame (such as the marginalized image frame), and a reference frame with a later acquisition time is used as the second reference frame (such as the image frame whose acquisition time is closest to the marginalized image frame and which shares the landmark point with the marginalized image frame), and the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame is updated to the inverse depth value based on the second reference frame, thereby updating the inverse depth parameters used by the preset visual positioning engine.
[0086] In some embodiments of the present application, the conversion of the reference frame of the landmark point corresponding to the marginalized image frame as the reference frame into other image frames in the image frame sequence includes: for each landmark point corresponding to the marginalized image frame as the reference frame, determining whether the other image frames in the image frame sequence include an image frame in which the landmark point is observed; in response to the other image frames not including the image frame in which the landmark point is observed, deleting the landmark point corresponding to the feature point.
[0087] Take the current moment as moment t, the marginalized image frame as the image frame captured at moment tn, and the landmark point L1 observed in the marginalized image frame as an example, that is, the marginalized image frame is the reference frame of the feature point p1 of the landmark point L1. For the landmark feature point p1 with this image frame as the reference frame, first determine the image frame that shares the landmark point L1 with the marginalized image frame in the image frames captured from moment t-n+1 to moment t (for example, determine the image frame that serves as the reference frame of the landmark feature point p1 in the image frames captured from moment t-n+1 to moment t). If there is no image frame that shares the landmark point L1 with the marginalized image frame, it means that the landmark point L1 has been lost, and the feature point corresponding to the landmark point is deleted.
[0088] During the marginalization process, the specific implementation method of updating the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame to the inverse depth value based on the second reference frame is described in the aforementioned steps and will not be repeated here.
[0089] The specific method of performing visual positioning based on the information of the updated landmark points corresponding to the feature points is a prior art and will not be described in detail in the embodiments of the present application.
[0090] The visual positioning method based on inverse depth parameters disclosed in the embodiment of the present application determines the corresponding feature points of the landmark points observed in the current image frame in response to obtaining the current image frame captured by the camera during the visual positioning based on the inverse depth parameters; determines whether the current image frame is a key frame; in response to the current image frame being a key frame, sets the reference frame of the newly added landmark points corresponding to the feature points observed in the current image frame relative to the previous key frame as the current image frame; in response to the current image frame not being a key frame, sets the reference frame of the newly added landmark points corresponding to the feature points observed in the current image frame relative to the previous image frame as the current key frame; updates the information of the landmark points corresponding to the feature points observed in the image frame sequence through the information of the landmark points corresponding to the feature points observed in the current image frame, so that the preset visual positioning engine positions the camera based on the updated information of the landmark points corresponding to the feature points, which helps to improve the accuracy of visual positioning.
[0091] The visual positioning method based on inverse depth parameters disclosed in the embodiment of the present application converts the reference frame of the feature point corresponding to the landmark point from the image frame in which the landmark point is first observed to the subsequent most recently acquired key frame with a common view relationship, or the ordinary image frame, when the key frame is collected. This allows the positioning system to retain the feature point information corresponding to the landmark point in a good tracking state when optimizing the landmark point data used for positioning, thereby maximizing the use of landmark point constraints to achieve the effect of improving positioning accuracy.
[0092] On the other hand, in the embodiment of the present application, the newly added landmark points observed in the non-key frames are also tracked and updated, which can further improve the positioning accuracy.
[0093] Embodiment 2
[0094] The embodiment of the present application discloses a visual positioning device based on inverse depth parameters, such as Figure 4 As shown, the device comprises:
[0095] The current image frame acquisition module 410 is used to determine the feature points corresponding to the landmark points observed in the current image frame in response to acquiring the current image frame captured by the camera during the visual positioning based on the inverse depth parameter;
[0096] A key frame determination module 420 is used to determine whether the current image frame is a key frame;
[0097] A first reference frame setting module 430 is used for setting the reference frame of the feature point corresponding to the newly added landmark point observed in the current image frame relative to the previous key frame as the current image frame in response to the current image frame being a key frame;
[0098] A second reference frame setting module 440 is configured to set, in response to the current image frame not being a key frame, a reference frame of a feature point corresponding to a newly added landmark point observed in the current image frame relative to a previous image frame as the current key frame;
[0099] The feature point information updating module 450 is used to update the information corresponding to the feature points of the landmark points observed in the image frame sequence through the information corresponding to the feature points of the landmark points observed in the current image frame, so that the preset visual positioning engine can position the camera based on the updated information corresponding to the feature points of the landmark points.
[0100] In some embodiments of the present application, the feature point information updating module 450 is further used to:
[0101] Adding the current image frame to the image frame sequence;
[0102] In response to the length of the image frame sequence reaching an upper threshold of marginalization processing, determining marginalized image frames in the image frame sequence;
[0103] The reference frame of the feature points corresponding to the landmark points with the marginalized image frame as the reference frame is converted into other image frames in the image frame sequence, wherein the other image frames refer to image frames in the image frame sequence except the marginalized image frame.
[0104] In some embodiments of the present application, converting the reference frame of the landmark point corresponding to the feature point using the marginalized image frame as the reference frame into other image frames in the image frame sequence includes:
[0105] For each landmark point corresponding to the marginalized image frame as a reference frame, determining whether other image frames in the image frame sequence include a key frame in which the landmark point is observed;
[0106] In response to the other image frames including the key frame in which the landmark point is observed, the operation of updating the information of the feature points corresponding to the landmark points observed in the image frame sequence is terminated.
[0107] In some embodiments of the present application, converting the reference frame of the landmark point corresponding to the feature point using the marginalized image frame as the reference frame into other image frames in the image frame sequence includes:
[0108] For each landmark point corresponding to the marginalized image frame as a reference frame, determining whether other image frames in the image frame sequence include a key frame in which the landmark point is observed;
[0109] In response to the fact that the other image frames do not include the key frame in which the landmark point is observed, determining the other image frame with the earliest acquisition time among the other image frames in which the landmark point is observed as the second reference frame, and using the marginalized image frame as the first reference frame;
[0110] The inverse depth value of the feature point corresponding to the landmark point based on the first reference frame is updated to the inverse depth value based on the second reference frame.
[0111] In some embodiments of the present application, converting the reference frame of the landmark point corresponding to the feature point using the marginalized image frame as the reference frame into other image frames in the image frame sequence includes:
[0112] For each landmark point corresponding to the marginalized image frame as a reference frame, determining whether other image frames in the image frame sequence include an image frame in which the landmark point is observed;
[0113] In response to the other image frames not including an image frame in which the landmark point is observed, the feature point corresponding to the landmark point is deleted.
[0114] In some embodiments of the present application, updating the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame to the inverse depth value based on the second reference frame includes:
[0115] The inverse depth value of the feature point corresponding to the landmark point based on the first reference frame is converted into an inverse depth value based on the second reference frame by using a preset inverse depth conversion function; wherein the preset inverse depth conversion function performs the following operations:
[0116] Determine the spatial position information of the feature point corresponding to the landmark point based on the second reference frame according to the spatial position of the feature point corresponding to the landmark point based on the first reference frame, the camera posture rotation matrix corresponding to the first reference frame, and the visual residual between the first reference frame and the second reference frame of the feature point corresponding to the landmark point;
[0117] According to the spatial position information of the feature point corresponding to the landmark point based on the second reference frame and the camera posture rotation matrix corresponding to the second reference frame, the inverse depth value of the feature point corresponding to the landmark point based on the second reference frame is determined.
[0118] In some embodiments of the present application, after updating the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame to the inverse depth value based on the second reference frame, the method further includes:
[0119] Determine the variance of the inverse depth value of the feature point corresponding to the landmark point based on the second reference frame according to the variance of the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame and the Jacobian matrix of the preset inverse depth conversion function on the inverse depth value;
[0120] The variance of the feature points corresponding to the landmark points in the preset visual positioning engine parameters is updated through the variance of the inverse depth value of the feature points corresponding to the landmark points based on the second reference frame.
[0121] The visual positioning device based on inverse depth parameters disclosed in the embodiment of the present application is used to implement the visual positioning method based on inverse depth parameters described in Example 1 of the present application. The specific implementation methods of each module of the device will not be repeated here, and reference can be made to the specific implementation methods of the corresponding steps in the method embodiment.
[0122] The visual positioning device based on inverse depth parameters disclosed in the embodiment of the present application determines the corresponding feature points of the landmark points observed in the current image frame in response to obtaining the current image frame captured by the camera during the visual positioning based on the inverse depth parameters; determines whether the current image frame is a key frame; in response to the current image frame being a key frame, sets the reference frame of the newly added landmark points corresponding to the feature points observed in the current image frame relative to the previous key frame as the current image frame; in response to the current image frame not being a key frame, sets the reference frame of the newly added landmark points corresponding to the feature points observed in the current image frame relative to the previous image frame as the current key frame; updates the information of the landmark points corresponding to the feature points observed in the image frame sequence through the information of the landmark points corresponding to the feature points observed in the current image frame, so that the preset visual positioning engine positions the camera based on the updated information of the landmark points corresponding to the feature points, which helps to improve the accuracy of visual positioning.
[0123] The visual positioning device based on inverse depth parameters disclosed in the embodiment of the present application converts the reference frame of the feature point corresponding to the landmark point from the image frame in which the landmark point is first observed to the subsequent most recently acquired key frame with a common view relationship, or the ordinary image frame, when the key frame is collected. This allows the positioning system to retain the feature point information corresponding to the landmark point in a good tracking state when optimizing the landmark point data used for positioning, thereby maximizing the use of landmark point constraints to achieve the effect of improving positioning accuracy.
[0124] On the other hand, in the embodiment of the present application, the newly added landmark points observed in the non-key frames are also tracked and updated, which can further improve the positioning accuracy.
[0125] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0126] The above is a detailed introduction to a visual positioning method and device based on inverse depth parameters provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
[0127] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0128] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the electronic device according to the embodiment of the present application. The present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.
[0129] For example, Figure 5An electronic device that can implement the method according to the present application is shown. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc. The electronic device traditionally includes a processor 510 and a memory 520 and a program code 530 stored on the memory 520 and can be run on the processor 510, and the processor 510 implements the method described in the above embodiment when executing the program code 530. The memory 520 can be a computer program product or a computer-readable medium. The memory 520 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. The memory 520 has a storage space 5201 for the program code 530 of the computer program for executing any method step in the above method. For example, the storage space 5201 for the program code 530 can include various computer programs for implementing the various steps in the above method respectively. The program code 530 is a computer-readable code. These computer programs can be read from one or more computer program products or written into the one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. The computer program includes a computer readable code, and when the computer readable code is executed on an electronic device, the electronic device is caused to execute the method according to the above embodiment.
[0130] The embodiment of the present application further discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the visual positioning method based on inverse depth parameters as described in the first embodiment of the present application are implemented.
[0131] Such a computer program product may be a computer-readable storage medium, which may have a computer-readable storage medium having a Figure 5 The memory 520 in the electronic device shown in the figure is similar to the storage segment, storage space, etc. The program code can be compressed and stored in the computer-readable storage medium in an appropriate form. The computer-readable storage medium is generally as shown in the reference Figure 6 The portable or fixed storage unit. Generally, the storage unit includes computer readable code 530', which is a code read by a processor. When the code is executed by the processor, each step in the method described above is implemented.
[0132] The term "one embodiment", "embodiment" or "one or more embodiments" herein means that a particular feature, structure or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present application. In addition, please note that the examples of the term "in one embodiment" here do not necessarily all refer to the same embodiment.
[0133] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.
[0134] In the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.
[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A visual positioning method based on inverse depth parameters, characterized in that: include: In the process of visual positioning based on the inverse depth parameter, in response to obtaining a current image frame captured by the camera, determining a feature point corresponding to a landmark point observed in the current image frame; Determine whether the current image frame is a key frame; In response to the current image frame being a key frame, setting a reference frame of a feature point corresponding to a newly added landmark point observed in the current image frame relative to a previous key frame as the current image frame; In response to the current image frame not being a key frame, setting a reference frame of a feature point corresponding to a newly added landmark point observed in the current image frame relative to a previous image frame as a current key frame; By using the information of the feature points corresponding to the landmark points observed in the current image frame, the information of the feature points corresponding to the landmark points observed in the image frame sequence is updated, so that the preset visual positioning engine positions the camera based on the updated information of the feature points corresponding to the landmark points; The step of updating the information of the feature points corresponding to the landmark points observed in the image frame sequence by using the information of the feature points corresponding to the landmark points observed in the current image frame comprises: Adding the current image frame to the image frame sequence; In response to the length of the image frame sequence reaching an upper threshold of marginalization processing, determining marginalized image frames in the image frame sequence; The reference frame of the feature points corresponding to the landmark points with the marginalized image frame as the reference frame is converted into other image frames in the image frame sequence, wherein the other image frames refer to image frames in the image frame sequence except the marginalized image frame.
2. The method according to claim 1, characterized in that: The step of converting the reference frame of the landmark point corresponding to the feature point using the marginalized image frame as the reference frame into other image frames in the image frame sequence comprises: For each landmark point corresponding to the marginalized image frame as a reference frame, determining whether other image frames in the image frame sequence include a key frame in which the landmark point is observed; In response to the other image frames including the key frame in which the landmark point is observed, the operation of updating the information of the feature points corresponding to the landmark points observed in the image frame sequence is terminated.
3. The method according to claim 1, characterized in that The step of converting the reference frame of the landmark point corresponding to the feature point using the marginalized image frame as the reference frame into other image frames in the image frame sequence comprises: For each landmark point corresponding to the marginalized image frame as a reference frame, determining whether other image frames in the image frame sequence include a key frame in which the landmark point is observed; In response to the fact that the other image frames do not include the key frame in which the landmark point is observed, determining the other image frame with the earliest acquisition time among the other image frames in which the landmark point is observed as the second reference frame, and using the marginalized image frame as the first reference frame; The inverse depth value of the feature point corresponding to the landmark point based on the first reference frame is updated to the inverse depth value based on the second reference frame.
4. The method according to claim 1, characterized in that: The step of converting the reference frame of the landmark point corresponding to the feature point using the marginalized image frame as the reference frame into other image frames in the image frame sequence comprises: For each landmark point corresponding to the marginalized image frame as a reference frame, determining whether other image frames in the image frame sequence include an image frame in which the landmark point is observed; In response to the other image frames not including an image frame in which the landmark point is observed, the feature point corresponding to the landmark point is deleted.
5. The method according to claim 3, characterized in that: The step of updating the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame to the inverse depth value based on the second reference frame comprises: The inverse depth value of the feature point corresponding to the landmark point based on the first reference frame is converted into an inverse depth value based on the second reference frame by using a preset inverse depth conversion function; wherein the preset inverse depth conversion function performs the following operations: Determine the spatial position information of the feature point corresponding to the landmark point based on the second reference frame according to the spatial position of the feature point corresponding to the landmark point based on the first reference frame, the camera posture rotation matrix corresponding to the first reference frame, and the visual residual between the first reference frame and the second reference frame of the feature point corresponding to the landmark point; According to the spatial position information of the feature point corresponding to the landmark point based on the second reference frame and the camera posture rotation matrix corresponding to the second reference frame, the inverse depth value of the feature point corresponding to the landmark point based on the second reference frame is determined.
6. The method according to claim 5, characterized in that After the step of updating the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame to the inverse depth value based on the second reference frame, the method further includes: Determine the variance of the inverse depth value of the feature point corresponding to the landmark point based on the second reference frame according to the variance of the inverse depth value of the feature point corresponding to the landmark point based on the first reference frame and the Jacobian matrix of the preset inverse depth conversion function on the inverse depth value; The variance of the feature points corresponding to the landmark points in the preset visual positioning engine parameters is updated through the variance of the inverse depth value of the feature points corresponding to the landmark points based on the second reference frame.
7. An electronic device comprising a memory, a processor, and a program code stored in the memory and executable on the processor, characterized in that: When the processor executes the program code, the visual positioning method based on inverse depth parameters described in any one of claims 1 to 6 is implemented.
8. A computer-readable storage medium having a program code stored thereon, characterized in that: When the program code is executed by a processor, the steps of the visual positioning method based on inverse depth parameters described in any one of claims 1 to 6 are implemented.
