Monocular vision tightly coupled methods, devices, computer-readable media, and electronic devices

Abstract

This application belongs to the field of computer vision positioning and navigation, and relates to a monocular vision tightly coupled method, device, computer-readable medium, and electronic device. The method includes: real-time acquisition of multiple image frames captured by the monocular camera and IMU information corresponding to the image frames and measured by the IMU; preprocessing the image frames and the IMU information to obtain feature tracking information and pre-integration information; performing nonlinear optimization based on the feature tracking information and the pre-integration information to obtain initialization parameters, and obtaining 3D map points corresponding to successfully tracked feature points in the image frames; updating a sliding window using Kalman filtering based on the initialization parameters and the 3D map points, wherein the updating of the sliding window includes external and internal updates, and the 3D map points are used to implement the external update. This application enables system initialization under dynamic conditions and improves the system's accuracy and robustness.

Description

Technical Field

[0001] This application belongs to the field of computer vision positioning and navigation technology, specifically relating to a monocular vision tight coupling method, a monocular vision tight coupling device, a computer-readable medium, and an electronic device. Background Technology

[0002] Augmented Reality (AR) technology is a technology that integrates virtual information with the real world. It can use hardware carriers such as mobile phones or glasses to integrate computer-generated virtual information into the real three-dimensional world.

[0003] During the fusion process, localization is required. Currently, the industry has conducted extensive research on mobile AR localization. Some solutions use pure cameras, leveraging optical flow tracking to achieve 2D plane localization, but this method lacks true scale information. Others employ VIO (Visual Inertial Odometry) localization, based on nonlinear optimization and camera-IMU fusion. However, the optimization and iteration process of this algorithm is computationally intensive, making it unusable on low- to mid-range phones. Furthermore, existing VIO algorithms require the phone to be stationary during initialization to estimate the initial state, which is then used to directly update the sliding window state. This method limits user accessibility and severely impacts the user experience. Moreover, directly using the initial state for IMU state updates can lead to significant localization errors or even tracking failures if the initial state is poor. Summary of the Invention

[0004] The purpose of this application is to provide a monocular vision tight coupling method, a monocular vision tight coupling device, a computer-readable medium, and an electronic device that can overcome the problems of initialization needing to be performed under static conditions and poor system accuracy and robustness in related technologies.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to a first aspect of the embodiments of this application, a monocular vision tightly coupled method is provided, applied to a positioning system including a monocular camera and an IMU. The method includes: acquiring in real time multiple image frames captured by the monocular camera and IMU information corresponding to the image frames and measured by the IMU; preprocessing the image frames and the IMU information to obtain feature tracking information and pre-integration information; performing nonlinear optimization based on the feature tracking information and the pre-integration information to obtain initialization parameters, and obtaining three-dimensional map points corresponding to successfully tracked feature points in the image frames; updating a sliding window using Kalman filtering based on the initialization parameters and the three-dimensional map points, wherein the update of the sliding window includes external updates and internal updates, and the three-dimensional map points are used to implement the external updates.

[0007] According to a second aspect of the embodiments of this application, a monocular vision tightly coupled device is provided, configured in a positioning system including a monocular camera and an IMU. The device includes: a preprocessing module, configured to acquire in real time multiple image frames captured by the monocular camera and IMU information corresponding to the image frames and measured by the IMU, and preprocess the image frames and the IMU information to obtain feature tracking information and pre-integration information; an initialization module, configured to perform nonlinear optimization based on the feature tracking information and the pre-integration information to obtain initialization parameters, and obtain three-dimensional map points corresponding to successfully tracked feature points in the image frames; and an update module, configured to perform Kalman filtering update on a sliding window based on the initialization parameters and the three-dimensional map points, wherein the update of the sliding window includes external update and internal update, and the three-dimensional map points are used to implement the external update.

[0008] In some embodiments of this application, based on the above technical solutions, the preprocessing module 610 is configured to: extract corner points from the image frame and perform visual tracking of the corner points based on optical flow tracing to obtain the feature tracking information; obtain sub-IMU information within the time interval corresponding to any two adjacent image frames, and pre-integrate the sub-IMU information to obtain the pre-integrated information.

[0009] In some embodiments of this application, based on the above technical solutions, the initialization module includes: a keyframe determination unit, used to determine image keyframes according to the feature tracking information; an information acquisition unit, used to acquire SFM information and pre-integration information to be decoupled corresponding to any two adjacent image keyframes; and a decoupling unit, used to decouple the pre-integration information to be decoupled according to the SFM information to obtain the initialization parameters.

[0010] In some embodiments of this application, based on the above technical solutions, the decoupling unit includes: a coordinate transformation unit, used to construct a coordinate transformation relationship between the camera coordinate system where the image frame is located and the IMU coordinate system where the IMU information is located; and a parameter determination unit, used to transform the pre-integration information to be decoupled based on the coordinate transformation relationship, and to decouple the transformed pre-integration information to be decoupled based on the SFM information to obtain the initialization parameters.

[0011] In some embodiments of this application, the initialization parameters include the gyroscope bias corresponding to the IMU; based on the above technical solutions, the parameter determination unit is configured to: transform the attitude information in the pre-integration information to be decoupled according to the coordinate transformation relationship to obtain target attitude information; determine the target attitude change amount according to the pre-integration information to be decoupled corresponding to two adjacent image keyframes; determine a first relational expression according to the target attitude information and the target attitude change amount, and optimize the first relational expression using the Gauss-Newton method to obtain the gyroscope bias.

[0012] In some embodiments of this application, the initialization parameters further include the velocity, scale factor, and gravity vector corresponding to each image frame; based on the above technical solution, the parameter determination unit is further configured to: construct a second relational expression containing the velocity, scale factor, and gravity vector according to the coordinate transformation relationship and the pre-integration information to be decoupled; optimize the second relational expression to obtain the velocity, scale factor, and gravity vector.

[0013] In some embodiments of this application, based on the above technical solutions, the initialization module is configured to: acquire all target image frames containing the same feature point, and perform triangulation processing according to the coordinates of the feature point in the target image frame to obtain the three-dimensional map point.

[0014] In some embodiments of this application, the initialized system includes an initialized monocular camera and an initialized IMU. Based on the above technical solution, the update module includes: a state vector determination unit, used to determine a state vector based on the initialization parameters; a state vector expansion unit, used to predict and expand the state vector according to the information collected by the initialized IMU and the initialized monocular camera, and to expand the covariance matrix according to the information collected by the initialized monocular camera; a triangulation unit, used to triangulate the feature points according to the historical camera state to obtain three-dimensional feature points; and a feature update unit, used to construct constraints on the historical camera state according to the three-dimensional feature points and the three-dimensional map points to update the predicted and expanded state vector.

[0015] In some embodiments of this application, based on the above technical solutions, the state vector amplification unit is configured to: predict the parameters in the state vector based on the acceleration and angular velocity measured by the initialized IMU; and obtain the camera state corresponding to the image frame captured by the initialized monocular camera; convert the camera state into a camera state in the IMU coordinate system, and amplify the predicted state vector based on the converted camera state.

[0016] In some embodiments of this application, based on the above technical solutions, the three-dimensional feature points are used to internally update the state of the sliding window, and the three-dimensional map points are used to externally update the state of the sliding window.

[0017] In some embodiments of this application, based on the above technical solutions, the update module is further configured to: determine the number of camera states within the sliding window; when the number of camera states exceeds a threshold, remove the camera state furthest from the current time and its corresponding covariance.

[0018] In some embodiments of this application, based on the above technical solutions, the monocular vision tightly coupled device is further configured to: when it is detected that the monocular camera cannot take pictures normally during the movement, perform pose prediction based on the IMU information obtained by the IMU through a deep learning model to achieve localization.

[0019] According to a third aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the monocular vision tightly coupled method as described above.

[0020] According to a fourth aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the monocular vision tightly coupled method as described above by executing the executable instructions.

[0021] In the technical solution provided in this application embodiment, feature extraction and feature tracking are performed on image frames captured by a monocular camera to obtain feature tracking information. Simultaneously, pre-integration is performed on the IMU information corresponding to each image frame to obtain pre-integration information. Based on this feature tracking information and pre-integration information, nonlinear optimization is performed to obtain initialization parameters, thereby initializing the positioning system. Furthermore, 3D map points corresponding to successfully tracked feature points after successful initialization can be obtained. Then, the sliding window can be updated based on the initialization parameters and 3D map points. The update of the sliding window state includes external and internal updates, with 3D map points used for external updates. This application enables initialization in a non-static hardware state; it also allows for external updates of the sliding window using 3D map points corresponding to successfully tracked feature points determined after successful initialization, while simultaneously updating the sliding window internally. This results in higher accuracy and stronger robustness for the positioning system, avoiding tracking failures for a period after successful initialization; and it also improves the user experience and user stickiness of the product.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 A schematic diagram of the system architecture of a system applying the monocular vision tight coupling method of the present application is shown.

[0025] Figure 2 The schematic diagram illustrates the steps of the monocular vision tight coupling method in this application.

[0026] Figure 3 A schematic diagram illustrating the process of obtaining initialization parameters in this application is shown.

[0027] Figure 4 A schematic diagram illustrating the process of obtaining gyroscope bias in this application is shown.

[0028] Figure 5 A schematic diagram illustrating the process of updating a sliding window in this application is shown.

[0029] Figure 6A schematic block diagram of the monocular vision tight coupling device in this application is shown.

[0030] Figure 7 A schematic diagram of a computer system architecture suitable for implementing the embodiments of this application is shown. Detailed Implementation

[0031] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0032] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0033] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0035] Figure 1 An exemplary system architecture block diagram illustrating the application of the technical solution of this application is shown schematically.

[0036] like Figure 1As shown, the system architecture 100 may include a terminal device 110, a monocular camera 120, an IMU 130 (Inertial Measurement Unit), a network 140, and a server 150, all housed within the terminal device 110. The terminal device 110 may be an electronic device such as a mobile phone, glasses, a drone, or a robot. The monocular camera 120 is used to capture images during the movement of the terminal device 110, acquiring image frames. The IMU 130 provides its own IMU information during the movement of the terminal device 110, including acceleration and angular velocity. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. The server 150 determines initialization parameters based on the image frames acquired by the monocular camera 120 and the acceleration and angular velocity provided by the IMU 130, and updates the sliding window using Kalman filtering to achieve precise positioning of the terminal device 110.

[0037] In one embodiment of this application, after acquiring an image frame, the monocular camera 120 in the terminal device 110 uploads the image frame to the server 150 in real time via the network 140. The acceleration and angular velocity of the IMU 130 are also uploaded to the server 150 in real time via the network 140. Upon receiving the image frame and the corresponding IMU information, the server 150 can perform feature extraction and feature tracking on the image frame acquired by the monocular camera 120. It performs pre-integration based on the IMU information within the time range corresponding to two adjacent image frames, and then performs nonlinear optimization based on the feature tracking information and pre-integration information to determine initialization parameters. The server then initializes the positioning system based on these initialization parameters and acquires the 3D map points corresponding to the successfully tracked feature points in the image frame. After successful initialization, the state within the sliding window and the corresponding covariance are updated using the successful initialization state, thereby enabling the positioning system after initialization and sliding window update to achieve accurate AR positioning.

[0038] In one embodiment of this application, the monocular vision tight coupling method in this embodiment can also be implemented by the terminal device 110. Specifically, the terminal device 110 performs feature extraction and feature tracking on the image frames captured by the monocular camera 120, pre-integrates the IMU information within the time range corresponding to two adjacent image frames, and performs nonlinear optimization based on the feature tracking information and pre-integration information to obtain initialization parameters and three-dimensional map points corresponding to the successfully tracked feature points in the image frames after successful initialization. Then, the state and covariance of the sliding window are updated based on the initialization parameters and three-dimensional map points to improve the stability of the positioning system and achieve accurate positioning.

[0039] Depending on the implementation requirements, the system architecture in this embodiment can have any number of terminal devices 110 and servers 150. For example, server 105 can be a server cluster composed of multiple servers. Correspondingly, it can also have any number of monocular cameras 120 and IMUs 130.

[0040] In some embodiments of this application, a deep learning model can be built in the terminal device 110 or the server 150 to perform pose calculation based on IMU information when the monocular camera cannot acquire effective image frames. The deep learning model is a model based on artificial intelligence technology.

[0041] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0042] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0043] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image annotation, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, and simultaneous localization and mapping (SLAM).

[0044] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0045] Taking the terminal device 110 as a smartphone as an example, in the relevant technologies of this application, the industry has made a lot of attempts at mobile phone AR positioning, which are mainly divided into three parts. The first part uses a pure camera solution to achieve 2D plane positioning effect by using optical flow tracking; the second part adopts VIO positioning based on nonlinear optimization of camera and IMU fusion; and the third part adopts the EKF method to perform VIO positioning by camera and IMU fusion.

[0046] However, each of these three parts has its own drawbacks. The first part of the method does not have real scale information. The second part of the method requires a lot of computing resources during the optimization and iteration process, making it unusable on low-end and mid-range mobile phones. The third part of the method requires initialization when the mobile phone is stationary, and even if the initialization is successful, there is still a risk of tracking failure in a short period of time.

[0047] The following detailed description of the monocular vision tight coupling method, monocular vision tight coupling device, computer-readable medium, and electronic device provided in this application, with reference to specific embodiments, provides a detailed explanation of these technical solutions.

[0048] Figure 2 This schematically illustrates a flowchart of a monocular vision tight coupling method in one embodiment of this application. This monocular vision tight coupling method can be executed by a terminal device or a server. Specifically, the terminal device or server can be... Figure 1 The terminal device 110 or server 150 in this application. The monocular vision tightly coupled method in this application is applied to a positioning system including a monocular camera and an IMU, such as... Figure 2 As shown, the monocular vision tight coupling method in this application embodiment can mainly include the following steps S210 to S230.

[0049] Step S210: Real-time acquisition of multiple image frames captured by the monocular camera and IMU information corresponding to the image frames and measured by the IMU, in order to obtain feature tracking information and pre-integration information;

[0050] Step S220: Perform nonlinear optimization based on the feature tracking information and the pre-integration information to obtain initialization parameters and obtain 3D map points corresponding to feature points in the image frame;

[0051] Step S230: Update the sliding window using Kalman filtering based on the initialization parameters and the 3D map points, wherein the update of the sliding window includes external updates and internal updates, and the 3D map points are used to implement the external updates.

[0052] In the technical solution provided in this application embodiment, feature extraction and feature tracking are performed on image frames captured by a monocular camera to obtain feature tracking information. Simultaneously, pre-integration is performed on the IMU information corresponding to each image frame to obtain pre-integration information. Based on this feature tracking information and pre-integration information, nonlinear optimization is performed to obtain initialization parameters, thereby initializing the positioning system. Furthermore, 3D map points corresponding to successfully tracked feature points after successful initialization can be obtained. Then, the sliding window can be updated based on the initialization parameters and 3D map points. The update of the sliding window state includes external and internal updates, with 3D map points used for external updates. This application enables initialization in a non-static hardware state; it also allows for external updates of the sliding window using 3D map points corresponding to successfully tracked feature points determined after successful initialization, while simultaneously updating the sliding window internally. This results in higher accuracy and stronger robustness for the positioning system, avoiding tracking failures for a period after successful initialization; and it also improves the user experience and user stickiness of the product.

[0053] The following section provides a detailed explanation of the specific implementation steps of the monocular vision tightly coupled method.

[0054] In step S210, multiple image frames captured by the monocular camera and IMU information corresponding to the image frames and measured by the IMU are acquired in real time to obtain feature tracking information and pre-integration information.

[0055] In one embodiment of this application, a VIO (Visual Inertial Odometry) algorithm is used to achieve AR positioning of the terminal device. The VIO algorithm utilizes information from the monocular camera and inertial measurement unit (IMU) on the terminal device to fuse computer-generated virtual information with real-world position and attitude information. After activating the positioning system in the terminal device, the monocular camera can be used to capture images of the scene within its shooting range, and positioning and navigation can be performed based on the captured video stream. However, since monocular cameras suffer from scale blur, the IMU can be used to provide motion state information to recover the scale information of the monocular vision.

[0056] In one embodiment of this application, the initialization of the positioning system is extremely important for the VIO algorithm. The determination of the initialization parameters determines the accuracy of the positioning system. Therefore, in this embodiment of the application, the positioning system is first initialized based on the video stream captured by the monocular camera and the IMU information provided by the IMU. Then, the state and covariance of the sliding window are updated based on the initialization state.

[0057] When initializing the positioning system, the video stream captured by the monocular camera and the corresponding IMU information measured by the IMU are first acquired in real time. Since the video stream contains multiple image frames, the initialization parameters are essentially determined based on these multiple image frames and their corresponding IMU information. After acquiring the multiple image frames and their corresponding IMU information, preprocessing can be performed to obtain feature tracking information and pre-integration information. The following section will explain in detail how to obtain feature tracking information and pre-integration information.

[0058] In one embodiment of this application, when preprocessing multiple image frames, feature points are first extracted from the image frames, and then the feature points are tracked using optical flow to establish constraints between consecutive frames. For example, feature points can be extracted from the starting image frame of multiple image frames arranged in chronological order, and then the detected feature points can be tracked in subsequent image frames using optical flow. If the number of feature points gradually decreases, the feature points can be extracted again and tracked.

[0059] In this application, the Fast corner detection algorithm is used to extract feature points from image frames. Specifically, firstly, a pixel P is randomly selected in the image frame, and a circle with a radius of 3 pixels is drawn to cover 16 pixels around point P. The pixel values ​​(brightness) of these 16 pixels are then compared with the pixel value (brightness) L of point P. P Compare the pixels; if there is a pixel value less than L... P -t or there are n consecutive pixels greater than L PIf the threshold value is +t, then this point is identified as a corner point, i.e., a feature point. Here, t is a threshold greater than 0, and n can be a positive integer greater than 4 and less than 16, such as 12. To improve the corner point detection rate, pixels numbered 1, 5, 9, and 12 out of the 16 pixels can be detected first, determining whether three of these four points have pixel values ​​(brightness) exceeding L. P If the condition is not met (+t), skip the step; if it is met, continue using the algorithm to determine if there are n pixels out of 16 that meet the condition. Pixels numbered 1 and 9 are pixels on the same vertical axis as point P, and pixels numbered 5 and 12 are pixels on the same horizontal axis as point P. To avoid detected corner points being adjacent to each other, non-maximum suppression can be used to filter the detected corner points. For example, a 3x3 or other size window can be created for any corner point Q. If another corner point appears in the window, compare the pixel values ​​(brightness) of corner point Q and the other corner point. If the pixel value (brightness) of corner point Q is larger, delete the other corner point; if the pixel value (brightness) of corner point Q is smaller, delete corner point Q.

[0060] In one embodiment of this application, 120 feature points can be extracted from an image frame. Of course, other numbers of feature points can also be extracted. This application does not specifically limit this.

[0061] In one embodiment of this application, after acquiring feature points, optical flow can be used to track the feature points and establish constraints between consecutive frames. Optical flow is a description of motion information of image brightness. This motion pattern refers to the apparent movement between an object, surface, edge, and background from a viewpoint by an observer (such as an eye, camera, etc.), evaluating the deformation between two images. In the embodiments of this application, the optical flow method used can be the KLT optical flow method, and other optical flow methods can also be used. This application does not specifically limit the specific methods used.

[0062] Feature point extraction and feature tracking can be performed using the methods described above to obtain feature tracking information. Simultaneously, preprocessing of the IMU's own IMU information is required. This preprocessing specifically involves acquiring sub-IMU information within the time interval corresponding to any two adjacent image frames and pre-integrating this sub-IMU information. The IMU information specifically includes the IMU's acceleration and angular velocity. By pre-integrating the IMU's acceleration and angular velocity between corresponding time points of two adjacent image frames, the state change between the two adjacent image frames can be obtained.

[0063] Set the acquisition time point t k and t k+1 Corresponding image frame b k and b k+1The corresponding changes in position, velocity, and attitude can be calculated through pre-integration, as shown in formula (1):

[0064]

[0065] in,

[0066] In the world coordinate system w, under b k+1 The position, velocity, and orientation of the image frame; In the world coordinate system w, under b k The image frame corresponds to the position, velocity, and attitude; α and ω represent the measured acceleration and angular velocity values ​​of the three degrees of freedom, respectively; b a and b ω These are acceleration bias and gyroscope bias, which are unknowns that need to be estimated; n a and n ω It consists of acceleration noise and gyroscope white noise, R w It is the current attitude of the IMU relative to the gravity vector in the world coordinate system.

[0067] It is worth noting that the typical image frequency is 30Hz, while the IMU frequency is 400Hz. Therefore, there is a lot of IMU information between the time points corresponding to two adjacent image frames. During pre-integration, it is necessary to process all the IMU information between the time points corresponding to two adjacent image frames.

[0068] In order to express the world coordinate system w under b as represented by formula (1) k+1 Transform the frame's position, velocity, and attitude into the IMU coordinate system b k In formula (1), both sides can be multiplied by the coordinates from the world coordinate system w to b. k Transformation of the IMU coordinate system corresponding to the frame and The specific expression is shown in formula (2):

[0069]

[0070] in,

[0071]

[0072]

[0073] The state change between two adjacent image frames can be determined by formula (1) or (2). Furthermore, the covariance matrix and Jacobian matrix of the pre-integration error can be obtained through further calculation.

[0074] In step S220, nonlinear optimization is performed based on the feature tracking information and the pre-integration information to obtain initialization parameters, and three-dimensional map points corresponding to the successfully tracked feature points in the image frame are obtained.

[0075] In one embodiment of this application, after obtaining feature tracking information and pre-integration information, nonlinear optimization initialization can be performed based on the feature tracking information and pre-integration information to obtain initialization parameters. Since the initialization is implemented using a nonlinear optimization method, initialization can be achieved under non-static conditions, eliminating the need for the user to ensure the terminal device is stationary when initially starting it, thus ensuring smooth initialization of the positioning system, greatly improving the user experience, and further enhancing user stickiness of the product.

[0076] Figure 3 A flowchart illustrating the process of obtaining initialization parameters is shown, such as... Figure 3 As shown, in step S301, image keyframes are determined based on the feature tracking information; in step S302, SFM information and pre-integration information to be decoupled are obtained corresponding to any two adjacent image keyframes; in step S303, the pre-integration information to be decoupled is decoupled based on the SFM information to obtain the initialization parameters.

[0077] Since SFM information is needed during initialization, and SFM information is obtained by processing image frames using the visual SFM algorithm, if two adjacent image frames are taken by the terminal device in a static state, then the parallax between the two adjacent image frames is zero. When processing with the visual SFM algorithm, the required SFM information cannot be obtained. Therefore, before obtaining SFM information, it is necessary to determine the image keyframes.

[0078] In step S301, image keyframes can be determined from multiple image frames based on disparity. The disparity between two adjacent image frames is obtained. If the disparity is greater than or equal to a disparity threshold, the image frame is an image keyframe; if the disparity is less than the disparity threshold, the image frame is not an image keyframe. Specifically, the disparity can be the optical flow length from one image frame to another corresponding to a feature point. Further, in this embodiment, image keyframes can also be determined based on the number of feature points in the image frame. When the number of feature points in an image frame is less than a preset threshold, that image frame is used as an image keyframe. By determining image keyframes and performing nonlinear optimization initialization based on them, data processing efficiency and the accuracy of initialization parameters can be improved. After determining the image keyframes, the image keyframes can be processed using a visual SFM algorithm to obtain SFM information. Simultaneously, the pre-integration information between two adjacent image keyframes can be determined according to the pre-integration calculation method in step S210. Then, the pre-integration information between two adjacent image keyframes is decoupled based on the SFM information to obtain initialization parameters. For ease of explanation, the pre-integration information between two adjacent keyframes of an image is referred to as the pre-integration information to be decoupled.

[0079] In one embodiment of this application, during nonlinear optimization initialization, since the coordinate system of the image frame is the camera coordinate system and the coordinate system of the IMU information is the IMU coordinate system, and the camera coordinate system is different from the IMU coordinate system, it is necessary to construct the coordinate transformation relationship between the camera coordinate system and the IMU coordinate system before initialization. Then, based on the coordinate transformation relationship, the pre-integration information to be decoupled is transformed, and the transformed pre-integration information to be decoupled is decoupled according to the SFM information to obtain the initialization parameters.

[0080] In constructing the coordinate transformation relationship, the camera coordinate system corresponding to the starting (frame 0) image frame is used as the initial camera coordinate system. Then, based on the coordinates of the image frame and the IMU coordinates, the IMU coordinates are transformed from the IMU coordinate system to the initial camera coordinate system. Alternatively, the camera coordinate system corresponding to the monocular camera can also be transformed to the IMU coordinate system.

[0081] Initialization parameters include the IMU's gyroscope bias and acceleration bias, as well as the position, velocity, attitude, and scale factor corresponding to each image frame. In the embodiments of this application, the gyroscope bias is first estimated, and then the position, velocity, attitude, and scale factor of each image frame are estimated. When estimating the gyroscope bias, the gyroscope bias is estimated based on the rotation relationship between multiple sets of two consecutive image keyframes obtained from monocular visual odometry and the transformation relationship between each set of adjacent image keyframes calculated by the IMU through pre-integration. Here, the rotation relationship is the pose relationship generated by processing adjacent two image keyframes through the visual SFM algorithm, and the transformation relationship is the pre-integration increment between adjacent two image keyframes. Multiple sets of rotation relationships and multiple sets of transformation relationships can strengthen the constraints and accurately estimate the gyroscope bias.

[0082] Figure 4 A flowchart illustrating the process of obtaining gyroscope bias is shown, as follows: Figure 4 As shown: In step S401, the attitude information in the pre-integration information to be decoupled is transformed according to the coordinate transformation relationship to obtain the target attitude information; in step S402, the target attitude change is determined according to the pre-integration information to be decoupled corresponding to the key frames of two adjacent images; in step S403, a first relation is determined according to the target attitude information and the target attitude change, and the first relation is optimized by the Gauss-Newton method to obtain the gyroscope deviation.

[0083] In step S401, the key frame b of the image is... k b k+1 Corresponding posture and The target attitude information is converted to the initial camera coordinate system based on the coordinate transformation relationship. and In step S402, based on image keyframe b k b k+1 Corresponding posture and It can also determine the target attitude change of the IMU from frame k to frame k+1. In step S403, the first relation can be determined based on the target attitude information determined in step S401 and the target attitude change determined in step S402. The parameter to be optimized in the first relation is the gyroscope bias of the IMU. The expression of the first relation is shown in formula (3):

[0084]

[0085] Where, δb w The optimization variable is the gyroscope bias, where k∈B indicates that there are B IMU measurements. With δb w Related, The information about δb can be obtained through a first-order approximation. w The equation is shown in formula (4):

[0086]

[0087] in, By optimizing formula (3) using the Gauss-Newton method, the gyroscope bias δb can be obtained. w .

[0088] In one embodiment of this application, a second relational expression containing velocity, scale factor, and gravity vector can be determined based on the IMU pre-integration formula (2) and the coordinate transformation relationship, as shown in formula (5):

[0089]

[0090]

[0091] in, It represents the pose of the k-th image frame relative to the 0-th image frame. It is the attitude calibration from IMU to camera. The k-th image frame does not have the correct scale position relative to the 0-th image frame. 's' is the position from the IMU to the camera, and 's' is the scale factor.

[0092] Considering that a set of constraints can be constructed from two consecutive keyframes of adjacent images, the second relation can be obtained according to formulas (2) and (5), as shown in formula (6):

[0093]

[0094]

[0095] By optimizing formula (6), all state variables can be obtained. in, The velocity corresponding to the i-th image frame can be determined based on the gyroscope deviation and state variables.

[0096] In one embodiment of this application, after the positioning system is successfully initialized according to the initialization parameters, the three-dimensional map points corresponding to the feature points can be obtained by triangulating the successfully tracked feature points in multiple image frames. Specifically, all target image frames containing the same feature point are obtained, and then triangulation is performed according to the coordinates of the feature point in the target image frame to obtain the three-dimensional map point of the feature point in the world coordinate system. The three-dimensional map point can be used to update the sliding window externally.

[0097] In step S230, the sliding window is updated using Kalman filtering based on the initialization parameters and the 3D map points. The update of the sliding window includes external updates and internal updates, and the 3D map points are used to implement the external updates.

[0098] In one embodiment of this application, after obtaining the initialization parameters, the system can be initialized according to the initialization parameters, and then the sliding window can be updated by Kalman filtering according to the successful initialization status and the 3D map points.

[0099] Figure 5 A flowchart illustrating the process of updating a sliding window is shown, such as... Figure 5 As shown, in step S501, the state vector is determined based on the initialization parameters; in step S502, the state vector is predicted and amplified based on the information collected by the initialized IMU and the initialized monocular camera, and the covariance matrix is ​​expanded based on the information collected by the initialized monocular camera; in step S503, the feature points are triangulated based on the historical camera state to obtain three-dimensional feature points; in step S504, constraints on the historical camera state are constructed based on the three-dimensional feature points and the three-dimensional map points to update the predicted and amplified state vector.

[0100] In step S501, after obtaining the initialization parameters, the system can be initialized according to the initialization parameters, and a state vector can be obtained. This state vector includes information in five dimensions: position, attitude, velocity, acceleration deviation, and gyroscope deviation. It is worth noting that the acceleration deviation is not estimated during the initialization phase; therefore, the acceleration deviation in the state vector is a preset value, such as 0 or other values. As the sliding window is continuously updated, the acceleration deviation is optimized to obtain the optimal acceleration deviation. Since acceleration deviation and gyroscope deviation affect the true acceleration and true angular velocity, and the true acceleration and true angular velocity affect the prediction of the state vector, thus affecting the update effect of the sliding window, the acceleration deviation can be adjusted according to the update effect of the sliding window. When the update effect of the sliding window reaches its best, the acceleration deviation is optimal.

[0101] In step S502, the acceleration and angular velocity measured by the initialized IMU are acquired in real time. The IMU state in the state vector can be predicted based on the acquired acceleration and angular velocity. The acceleration and angular velocity measured by the IMU are both measured values. Due to the effects of acceleration bias and gyroscope bias, the measured values ​​have deviations. The true value is the value after correcting the measured value according to the deviation, that is, true value = measured value - deviation. The prediction of each parameter in the state vector can be achieved by integrating the acceleration and angular velocity. This integration is the same as the pre-integration method in step S210. According to the corresponding camera state of the image frame acquired in real time by the initialized monocular camera, after acquiring the current camera state corresponding to the real-time acquired image frame, the current camera state can be converted into the state in the IMU coordinate system according to the coordinate transformation relationship between the camera coordinate system and the IMU coordinate system. The converted state is then added to the state vector to expand the state vector. It is worth noting that while pre-integrating the IMU information, the covariance matrix corresponding to the IMU information can be obtained. Therefore, while expanding the state vector according to the current camera state, the covariance matrix can also be expanded to add the covariance between the IMU and the monocular camera, as well as the variance of the monocular camera itself, to the initial variance corresponding to the IMU, thus forming an expanded covariance matrix.

[0102] In step S503, all historical image frames before the current time point are obtained. Each historical image frame corresponds to a historical camera state. By extracting and tracking the feature points in each historical image frame, the target historical image frame containing the same two-dimensional feature point can be determined. Then, triangulation estimation can be performed based on the historical camera state corresponding to the target historical image frame to obtain the three-dimensional feature point corresponding to the two-dimensional feature point. For example, 10 historical image frames containing the same feature point can be obtained for triangulation estimation.

[0103] In step S504, constraints on the states of multiple historical cameras can be constructed based on the 3D map points determined after successful external initialization and the 3D feature points determined by internal triangulation estimation, thereby updating the state vector.

[0104] In one embodiment of this application, 3D map points are used for external updates of the sliding window, and 3D feature points are used for internal updates. Since the internal updates of the sliding window based on 3D feature points are highly dependent on the historical camera state corresponding to the acquired historical image frames, if the initial IMU integration is inaccurate or fluctuates significantly, the historical camera state will also be biased, leading to significant deviations in the internal updates. However, since the 3D map points are determined based on the successfully initialized system, their accuracy is higher. Therefore, using 3D map points to update the sliding window can prevent update failure due to significant deviations in the internal updates, thereby improving the robustness and accuracy of the positioning system. In other words, simultaneous internal and external updates of the sliding window can improve the accuracy of the positioning system, while the external update plays a decisive role in the success of the sliding window update, resulting in higher accuracy and robustness. The positioning system after successful initialization and sliding window updates can accurately estimate the camera pose, achieving precise positioning and navigation. AR positioning using the positioning system obtained through the technical solution of this application shows an accuracy improvement of over 20% compared to existing AR positioning systems, and can resolve over 80% of tracking loss phenomena.

[0105] In one embodiment of this application, since a certain number of camera states are constrained within the sliding window, when updating the sliding window, it is also necessary to determine the number of camera states within the sliding window. When the number of camera states exceeds a threshold, the oldest camera state and its corresponding covariance are removed. In this embodiment of the application, the number of camera states in the sliding window can be set to 10, or it can be set to other values. This embodiment of the application does not specifically limit this.

[0106] In one embodiment of this application, during the operation of the terminal device, there may be situations where the monocular camera is blocked by obstacles or strong light, making it impossible to extract features from the captured image frames. Thus, it is impossible to perform positioning and navigation based on the image frames and IMU information. To address this situation, this embodiment of the application proposes to use a pre-trained deep learning model to predict the pose based on the IMU information. The terminal device performs positioning and navigation based on the predicted pose to achieve pure IMU positioning. When the monocular camera resumes normal shooting, it is then adjusted based on the predicted pose.

[0107] In the technical solution provided in this application embodiment, feature extraction and feature tracking are performed on image frames captured by the camera to obtain feature tracking information. Simultaneously, pre-integration is performed on the IMU information corresponding to each image frame to obtain pre-integration information. Based on this feature tracking information and pre-integration information, nonlinear optimization is performed to obtain initialization parameters, thereby initializing the system and acquiring successfully initialized 3D map points. Then, the state of the sliding window is updated based on the initialization parameters and 3D map points. The update of the sliding window state includes external and internal updates, with successfully initialized 3D map points used for external updates. This application enables initialization in a non-static hardware state; it also allows for external updates of the sliding window using 3D map points determined after successful initialization while simultaneously performing internal updates. This results in higher accuracy and stronger robustness after successful initialization, avoiding tracking failures for a period after successful initialization. Furthermore, it improves the user experience and user stickiness of the product.

[0108] The monocular vision tightly coupled method in this application is based on a VIO algorithm that combines nonlinear optimization and filtering. In other words, this application performs nonlinear optimization initialization within a filtering framework. Compared to existing technologies that only use nonlinear optimization or filtering methods for navigation and positioning, it offers higher accuracy and robustness, lower computational resource requirements, and can perform initialization under dynamic conditions, avoiding the limitation of existing initialization methods that rely on static conditions. Furthermore, when updating the sliding window using Kalman filtering, it simultaneously employs internal updates based on 3D feature points obtained through internal triangulation and external updates based on 3D map points determined after successful external initialization. This results in higher system accuracy and robustness, more accurate positioning, and solves the problem of large errors and easy tracking failures within a short period after successful initialization.

[0109] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0110] The following describes an embodiment of the device of this application. The monocular vision tight coupling device is configured in a positioning system including a monocular camera and an IMU, and can be used to execute the monocular vision tight coupling method in the above embodiments of this application. Figure 6 A schematic block diagram of a monocular vision tightly coupled device provided in an embodiment of this application is shown. Figure 6As shown, the monocular vision tightly coupled device 600 includes: a preprocessing module 610, an initialization module 620, and an update module 630, specifically:

[0111] The preprocessing module 610 is used to acquire multiple image frames captured by the monocular camera and IMU information corresponding to the image frames and measured by the IMU in real time, and to preprocess the image frames and the IMU information to obtain feature tracking information and pre-integration information; the initialization module 620 is used to perform nonlinear optimization based on the feature tracking information and the pre-integration information to obtain initialization parameters, and to obtain 3D map points corresponding to the successfully tracked feature points in the image frames; the update module 630 is used to update the sliding window using Kalman filtering based on the initialization parameters and the 3D map points, wherein the update of the sliding window includes external update and internal update, and the 3D map points are used to implement the external update.

[0112] In some embodiments of this application, based on the above technical solutions, the preprocessing module 610 is configured to: extract corner points from the image frame and perform visual tracking of the corner points based on optical flow tracing to obtain the feature tracking information; obtain sub-IMU information within the time interval corresponding to any two adjacent image frames, and pre-integrate the sub-IMU information to obtain the pre-integrated information.

[0113] In some embodiments of this application, based on the above technical solutions, the initialization module 620 includes: a keyframe determination unit, used to determine image keyframes according to the feature tracking information; an information acquisition unit, used to acquire SFM information and pre-integration information to be decoupled corresponding to any two adjacent image keyframes; and a decoupling unit, used to decouple the pre-integration information to be decoupled according to the SFM information to obtain the initialization parameters.

[0114] In some embodiments of this application, based on the above technical solutions, the decoupling unit includes: a coordinate transformation unit, used to construct a coordinate transformation relationship between the camera coordinate system where the image frame is located and the IMU coordinate system where the IMU information is located; and a parameter determination unit, used to transform the pre-integration information to be decoupled based on the coordinate transformation relationship, and to decouple the transformed pre-integration information to be decoupled based on the SFM information to obtain the initialization parameters.

[0115] In some embodiments of this application, the initialization parameters include the gyroscope bias corresponding to the IMU; based on the above technical solutions, the parameter determination unit is configured to: transform the attitude information in the pre-integration information to be decoupled according to the coordinate transformation relationship to obtain target attitude information; determine the target attitude change amount according to the pre-integration information to be decoupled corresponding to two adjacent image keyframes; determine a first relational expression according to the target attitude information and the target attitude change amount, and optimize the first relational expression using the Gauss-Newton method to obtain the gyroscope bias.

[0116] In some embodiments of this application, the initialization parameters further include the velocity, scale factor, and gravity vector corresponding to each image frame; based on the above technical solution, the parameter determination unit is further configured to: construct a second relational expression containing the velocity, scale factor, and gravity vector according to the coordinate transformation relationship and the pre-integration information to be decoupled; optimize the second relational expression to obtain the velocity, scale factor, and gravity vector.

[0117] In some embodiments of this application, based on the above technical solutions, the initialization module 620 is configured to: acquire all target image frames containing the same feature point, and perform triangulation processing according to the coordinates of the feature point in the target image frame to obtain the three-dimensional map point.

[0118] In some embodiments of this application, the initialized system includes an initialized monocular camera and an initialized IMU. Based on the above technical solutions, the update module 630 includes: a state vector determination unit, used to determine a state vector based on the initialization parameters; a state vector expansion unit, used to predict and expand the state vector according to the information collected by the initialized IMU and the initialized monocular camera, and to expand the covariance matrix according to the information collected by the initialized monocular camera; a triangulation unit, used to triangulate the feature points according to the historical camera state to obtain three-dimensional feature points; and a feature update unit, used to construct constraints on the historical camera state according to the three-dimensional feature points and the three-dimensional map points to update the predicted and expanded state vector.

[0119] In some embodiments of this application, based on the above technical solutions, the state vector amplification unit is configured to: predict the parameters in the state vector based on the acceleration and angular velocity measured by the initialized IMU; and obtain the camera state corresponding to the image frame captured by the initialized monocular camera; convert the camera state into a camera state in the IMU coordinate system, and amplify the predicted state vector based on the converted camera state.

[0120] In some embodiments of this application, based on the above technical solutions, the three-dimensional feature points are used to internally update the state of the sliding window, and the three-dimensional map points are used to externally update the state of the sliding window.

[0121] In some embodiments of this application, based on the above technical solutions, the update module 630 is further configured to: determine the number of camera states in the sliding window; when the number of camera states exceeds a threshold, remove the camera state furthest from the current time and its corresponding covariance.

[0122] In some embodiments of this application, based on the above technical solutions, the monocular vision tightly coupled device 600 is further configured to: when it is detected that the monocular camera cannot take pictures normally during the movement, perform pose prediction based on the IMU information obtained by the IMU through a deep learning model to achieve localization.

[0123] The specific details of the monocular vision tight coupling device provided in the various embodiments of this application have been described in detail in the corresponding method embodiments, and will not be repeated here.

[0124] Figure 7 This schematically illustrates a computer system architecture block diagram for an electronic device used to implement embodiments of the present application. The electronic device may be, for example... Figure 1 The terminal device 110, monocular camera 120, or IMU 130 shown are illustrated.

[0125] It should be noted that, Figure 7 The computer system 700 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0126] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for system operation. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output interface 705 (I / O interface) is also connected to the bus 704.

[0127] In some embodiments, the following components are connected to the input / output interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a local area network card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0128] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit 701, it performs various functions defined in the system of this application.

[0129] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable medium, or any combination of the above. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0132] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, and includes several instructions to cause an electronic device to execute the method according to the embodiments of this application.

[0133] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0134] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A monocular vision tightly coupled method, applied to a positioning system including a monocular camera and an IMU, characterized in that, include: Multiple image frames captured by the monocular camera and IMU information corresponding to the image frames and measured by the IMU are acquired in real time, and the image frames and the IMU information are preprocessed to obtain feature tracking information and pre-integration information; Nonlinear optimization is performed based on the feature tracking information and the pre-integration information to obtain initialization parameters and to obtain 3D map points corresponding to the feature points that were successfully tracked in the image frame. The sliding window is updated using Kalman filtering based on the initialization parameters and the 3D map points, wherein the step of updating the sliding window using Kalman filtering based on the initialization parameters and the 3D map points includes: Determine the state vector based on the initialization parameters; The state vector is predicted and expanded based on the information collected by the initialized IMU and the initialized monocular camera, and the covariance matrix is ​​expanded based on the information collected by the initialized monocular camera. The feature points are triangulated based on the historical camera status to obtain three-dimensional feature points. Constraints on the historical camera state are constructed based on the three-dimensional feature points and the three-dimensional map points to update the predicted and augmented state vector. The update of the sliding window includes external updates and internal updates. The three-dimensional map points are used to implement the external updates, and the three-dimensional feature points are used to implement the internal updates.

2. The method according to claim 1, characterized in that, The preprocessing of the image frame and the IMU information to obtain feature tracking information and pre-integration information includes: Corner points are extracted from the image frame, and the corner points are visually tracked based on optical flow tracing to obtain the feature tracking information; Obtain sub-IMU information within the time interval corresponding to any two adjacent image frames, and pre-integrate the sub-IMU information to obtain the pre-integrated information.

3. The method according to claim 1, characterized in that, The step of performing nonlinear optimization based on the feature tracking information and the pre-integration information to obtain initialization parameters includes: Image keyframes are determined based on the feature tracking information; Obtain the SFM information and the pre-integration information to be decoupled corresponding to any two adjacent keyframes of the image; The pre-integration information to be decoupled is decoupled based on the SFM information to obtain the initialization parameters.

4. The method according to claim 3, characterized in that, The step of decoupling the pre-integration information to be decoupled based on the SFM information to obtain the initialization parameters includes: Construct the coordinate transformation relationship between the camera coordinate system where the image frame is located and the IMU coordinate system where the IMU information is located; The pre-integration information to be decoupled is transformed based on the coordinate transformation relationship, and the transformed pre-integration information is decoupled according to the SFM information to obtain the initialization parameters.

5. The method according to claim 4, characterized in that, The initialization parameters include the gyroscope deviation corresponding to the IMU; The process of transforming the pre-integral information to be decoupled based on the coordinate transformation relationship, and decoupling the transformed pre-integral information according to the SFM information to obtain the initialization parameters, includes: The attitude information in the pre-integrated information to be decoupled is transformed according to the coordinate transformation relationship to obtain the target attitude information; The target pose change is determined based on the pre-integration information to be decoupled corresponding to two adjacent keyframes of the image. A first relation is determined based on the target attitude information and the target attitude change, and the first relation is optimized using the Gauss-Newton method to obtain the gyroscope deviation.

6. The method according to claim 4, characterized in that, The initialization parameters also include the velocity, scale factor, and gravity vector corresponding to each image frame; The process of transforming the pre-integral information to be decoupled based on the coordinate transformation relationship, and decoupling the transformed pre-integral information according to the SFM information to obtain the initialization parameters, includes: Based on the coordinate transformation relationship and the pre-integration information to be decoupled, a second relational expression is constructed that includes the velocity, the scale factor, and the gravity vector. The second relationship is optimized to obtain the velocity, the scale factor, and the gravity vector.

7. The method according to claim 1, characterized in that, The step of obtaining the 3D map points corresponding to the successfully tracked feature points in the image frame includes: All target image frames containing the same feature point are acquired, and triangulation is performed based on the coordinates of the feature point in the target image frames to obtain the three-dimensional map point.

8. The method according to claim 1, characterized in that, The initialized positioning system includes an initialized monocular camera and an initialized IMU.

9. The method according to claim 1, characterized in that, The step of predicting and expanding the state vector based on the information collected by the initialized IMU and the initialized camera includes: Predict the parameters in the state vector based on the acceleration and angular velocity measured by the initialized IMU; and Obtain the camera state corresponding to the image frames captured by the initialized monocular camera; The camera state is converted into a camera state in the IMU coordinate system, and the predicted state vector is amplified based on the converted camera state.

10. The method according to claim 1, characterized in that, The three-dimensional feature points are used to internally update the state of the sliding window, and the three-dimensional map points are used to externally update the state of the sliding window.

11. The method according to claim 1, characterized in that, The step of updating the sliding window using Kalman filtering based on the initialization parameters and the 3D map points further includes: The number of camera states within the sliding window is determined. When the number of camera states exceeds a threshold, the camera state furthest from the current time and its corresponding covariance are removed.

12. The method according to claim 1, characterized in that, The method further includes: When it is detected that the monocular camera cannot take pictures normally during movement, the pose prediction is performed by a deep learning model based on the IMU information obtained by the IMU to achieve localization.

13. A monocular vision tightly coupled device, configured in a positioning system comprising a monocular camera and an IMU, characterized in that, include: The preprocessing module is used to acquire multiple image frames captured by the monocular camera and IMU information corresponding to the image frames and measured by the IMU in real time, and to preprocess the image frames and the IMU information to obtain feature tracking information and pre-integration information. An initialization module is used to perform nonlinear optimization based on the feature tracking information and the pre-integration information to obtain initialization parameters and to obtain three-dimensional map points corresponding to feature points in the image frame. The update module is used to update the sliding window based on the initialization parameters and the 3D map points; The update module includes: A state vector determination unit is used to determine a state vector based on the initialization parameters; The state vector augmentation unit is used to predict and augment the state vector based on the information collected by the initialized IMU and the initialized monocular camera, and to augment the covariance matrix based on the information collected by the initialized monocular camera. A triangulation unit is used to triangulate the feature points according to the historical camera state to obtain three-dimensional feature points. The feature update unit is used to construct constraints on the historical camera state based on the three-dimensional feature points and the three-dimensional map points, so as to update the predicted and augmented state vector. The update of the sliding window includes external update and internal update. The three-dimensional map points are used to implement the external update, and the three-dimensional feature points are used to implement the internal update.

14. A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the monocular vision tightly coupled method according to any one of claims 1 to 12.

15. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the monocular vision tightly coupled method of any one of claims 1 to 12 by executing the executable instructions.

16. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable medium, and a processor of a computer device reads from the computer-readable medium and executes the computer program, causing the computer device to perform the monocular vision tightly coupled method according to any one of claims 1 to 12.

Images