Neural network-based binocular vision inertial tight coupling slam method and device

Abstract

This application discloses a binocular vision-inertial tightly coupled SLAM method and apparatus based on neural networks. The method includes: acquiring a time-synchronized first image sequence and a second image sequence; inputting the first image sequence and the second image sequence into a pre-trained neural network to obtain a first 3D point map and a first descriptor corresponding to the first image sequence, and a second 3D point map and a second descriptor corresponding to the second image sequence; performing spatiotemporal 3D matching on the first 3D point map, the first descriptor, the second 3D point map, and the second descriptor to obtain a set of matched 3D point pairs; constructing a visual constraint factor based on the set of matched 3D point pairs, and fusing the visual constraint factor with an IMU pre-integration factor model to obtain a tightly coupled factor model; solving the tightly coupled factor model using a nonlinear optimizer to obtain robot state estimation information and / or a dense 3D point cloud map. It can achieve high-precision, high-fidelity, and robust real-time localization and map construction.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and in particular relates to a binocular vision-inertial tightly coupled SLAM method and device based on neural networks. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a core technology that enables robots to achieve autonomous perception and movement in unknown environments, and it is widely used in fields such as drones, autonomous vehicles, and mobile robots. Among them, visual SLAM uses cameras as the main sensor to achieve localization and mapping through image information. It has advantages such as low cost and rich information, and has become a current research hotspot.

[0003] Traditional geometric-based visual SLAM methods include feature-point methods (such as VINS-Mono and ORB-SLAM3) and direct methods (such as DSO and DM-VIO). Feature-point methods extract and match sparse feature points in the image and optimize reprojection errors using multi-view geometry principles; direct methods bypass feature extraction and directly construct error functions based on the photometric consistency of image pixels. However, both methods heavily rely on the visual appearance of the environment: feature-point methods are prone to failure in low-texture environments due to feature loss; direct methods are highly sensitive to image blurring caused by changes in illumination and rapid motion. Furthermore, monocular camera-based systems inherently suffer from scale uncertainty. Although scale can theoretically be recovered through fusion with an inertial measurement unit (IMU), the initialization process is sensitive, and scale estimation is prone to drift under insufficient excitation.

[0004] In recent years, the emergence of "feedforward point map regression" models based on the Transformer architecture (such as DUSt3R and MASt3R) has marked a shift in the research paradigm of visual SLAM. These models can directly regress dense 3D point maps from images without requiring known camera intrinsic parameters. Leveraging large-scale 3D spatial priors learned from massive datasets, they achieve pixel-level dense matching and scene geometry reconstruction across viewpoints, fundamentally overcoming the reliance of traditional methods on manual feature extraction and explicit geometric verification, and demonstrating outstanding robustness.

[0005] However, to compensate for scale limitations, current Transformer-based fusion schemes (such as MASt3R-Fusion) attempt to fuse monocular Transformer models with IMUs. However, their fusion path still follows the traditional approach: first, the relative poses in the Sim(3) (3D similarity transformation space) between frames are estimated through post-processing steps such as point map alignment; then, this pose is used as a constraint factor for joint optimization with the IMU pre-integration factor. This "first calculate the pose, then construct the constraints" approach treats the Transformer as a "black box" pose provider, failing to directly utilize its core output—the geometrically rich 3D point map itself—leading to information waste and introducing unnecessary computational steps and linearization errors. Therefore, designing an inertial navigation framework that can directly and deeply fuse with its 3D point map output has become a key challenge in promoting the practical application of this new paradigm. Summary of the Invention

[0006] This application aims to address at least one of the technical problems existing in the related art. To this end, this application proposes a binocular vision-inertial tightly coupled SLAM method and device based on neural networks, which achieves high-precision, high-fidelity, and robust real-time localization and map construction.

[0007] In a first aspect, this application provides a binocular vision-inertial tightly coupled SLAM method based on neural networks, the method comprising: A first image sequence and a second image sequence synchronized with time are acquired. The first image sequence and the second image sequence are input into a pre-trained neural network to obtain a first three-dimensional point map and a first descriptor corresponding to the first image sequence, and a second three-dimensional point map and a second descriptor corresponding to the second image sequence. Spatiotemporal 3D matching is performed on the first 3D point map, the first descriptor, the second 3D point map, and the second descriptor to obtain a set of matched 3D point pairs, wherein the set of matched 3D point pairs includes at least one pair of matched 3D point pairs. Based on the set of matched 3D point pairs, a visual constraint factor is constructed, and the visual constraint factor is fused with the IMU pre-integration factor model to obtain a tightly coupled factor model. The tight coupling factor model is solved using a nonlinear optimizer to obtain the robot's state estimation information and / or a dense 3D point cloud map.

[0008] Secondly, this application provides a binocular vision-inertial tightly coupled SLAM device based on a neural network, the device comprising: The acquisition module is used to acquire a time-synchronized first image sequence and a second image sequence, input the first image sequence and the second image sequence into a pre-trained neural network, and obtain a first three-dimensional point map and a first descriptor corresponding to the first image sequence, as well as a second three-dimensional point map and a second descriptor corresponding to the second image sequence. The matching module is used to perform spatiotemporal three-dimensional matching of the first three-dimensional point map, the first descriptor, the second three-dimensional point map, and the second descriptor to obtain a set of matched three-dimensional point pairs, wherein the set of matched three-dimensional point pairs includes at least one pair of matched three-dimensional point pairs. The fusion module is used to construct a visual constraint factor based on the set of matched 3D point pairs, and fuse the visual constraint factor with the IMU pre-integration factor model to obtain a tightly coupled factor model; The solver module is used to solve the tightly coupled factor model using a nonlinear optimizer to obtain the robot's state estimation information and / or a dense 3D point cloud map.

[0009] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the neural network-based binocular vision-inertial tightly coupled SLAM method as described in the first aspect above.

[0010] Fourthly, this application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the neural network-based binocular vision-inertial tightly coupled SLAM method as described in the first aspect above.

[0011] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the neural network-based binocular vision-inertial tightly coupled SLAM method as described in the first aspect.

[0012] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the neural network-based binocular vision-inertial tightly coupled SLAM method as described in the first aspect above.

[0013] The above-described one or more technical solutions in the embodiments of this application have at least the following technical effects: The binocular vision-inertial tightly coupled SLAM method and device based on neural networks provided in this application directly regresses a 3D point map with a metric scale through neural networks, and comprehensively utilizes 3D information in both spatial and temporal dimensions for matching to obtain at least one pair of matched 3D point points. This provides a stable, accurate, and scaled geometric constraint foundation for subsequent optimization. Furthermore, the Hessian information matrix and residual vector of the matched 3D point pairs with respect to the system state variables are constructed using these pairs and encapsulated as a new visual constraint factor. This visual constraint factor and IMU factor are integrated into a unified, metric-scale tightly coupled factor graph model, and joint state estimation is performed through nonlinear optimization. This completely avoids the indirect process of first estimating relative pose, realizing the fusion of visual geometric information and inertial measurement information at the optimization level, improving accuracy and robustness, and achieving high-precision, high-fidelity, and highly robust real-time positioning and map construction.

[0014] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0015] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts of the binocular vision-inertial tightly coupled SLAM method based on neural networks provided in the embodiments of this application; Figure 2 This is the second flowchart of the binocular vision-inertial tightly coupled SLAM method based on neural networks provided in the embodiments of this application; Figure 3 This is the third flowchart of the binocular vision-inertial tightly coupled SLAM method based on neural networks provided in the embodiments of this application; Figure 4 This is the fourth flowchart of the binocular vision-inertial tightly coupled SLAM method based on neural networks provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the device for the binocular vision-inertial tightly coupled SLAM method based on neural networks provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0017] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0018] The following describes the relevant technologies involved in this application: Simultaneous Localization and Mapping (SLAM) is a core technology that enables robots to achieve autonomous perception and movement in unknown environments, and it is widely used in fields such as drones, autonomous vehicles, and mobile robots. Among them, visual SLAM uses cameras as the main sensor to achieve localization and mapping through image information. It has advantages such as low cost and rich information, and has become a current research hotspot.

[0019] Traditional geometric-based visual SLAM methods include feature-point methods (such as VINS-Mono and ORB-SLAM3) and direct methods (such as DSO and DM-VIO). Feature-point methods extract and match sparse feature points in the image and optimize reprojection errors using multi-view geometry principles; direct methods bypass feature extraction and directly construct error functions based on the photometric consistency of image pixels. However, both methods heavily rely on the visual appearance of the environment: feature-point methods are prone to failure in low-texture environments due to feature loss; direct methods are highly sensitive to image blurring caused by changes in illumination and rapid motion. Furthermore, monocular camera-based systems inherently suffer from scale uncertainty. Although scale can theoretically be recovered through fusion with an inertial measurement unit (IMU), the initialization process is sensitive, and scale estimation is prone to drift under insufficient excitation.

[0020] In recent years, the emergence of "feedforward point map regression" models based on the Transformer architecture (such as DUSt3R and MASt3R) has marked a paradigm shift in visual SLAM research. These models can directly regress dense 3D point maps from images without requiring known camera intrinsic parameters. Leveraging large-scale 3D spatial priors learned from massive datasets, they achieve pixel-level dense matching and scene geometry reconstruction across viewpoints, fundamentally overcoming the reliance on manual feature extraction and explicit geometric verification in traditional methods, demonstrating superior robustness. However, this emerging paradigm currently faces the following significant limitations in its early stages of development: 1. Limitations of Model Architecture: Existing Transformer vision models (such as DUSt3R, MASt3R) and their derived SLAM systems (such as MASt3R-SLAM) are mainly based on monocular vision design. This leads to the inheritance of the inherent defects of monocular systems: scale uncertainty. The output point map and pose exist in a Sim(3) space with arbitrary scale, and cannot directly provide measurement information.

[0021] 2. Shortcomings of Transformer-based fusion schemes: To compensate for scale defects, current schemes (such as MASt3R-Fusion) attempt to fuse monocular Transformer models with IMUs. However, their fusion path still follows the traditional approach: first, the relative poses of the Sim(3) between frames are estimated through post-processing steps such as point map alignment, and then the poses are used as constraint factors and jointly optimized with IMU pre-integration factors. This "first calculate the pose, then construct the constraints" approach treats the Transformer as a "black box" pose provider, failing to directly utilize its core output—the geometrically rich 3D point map itself—leading to information waste and introducing unnecessary computational steps and linearization errors.

[0022] Therefore, although the Transformer model brings powerful perceptual priors to SLAM, how to endow it with stable metric scale perception capabilities and design an inertial navigation framework that can be directly and deeply integrated with its 3D point map output has become a key challenge to promote the practical application of this new paradigm.

[0023] The following description, in conjunction with the accompanying drawings, details the neural network-based binocular vision-inertial tightly coupled SLAM method, neural network-based binocular vision-inertial tightly coupled SLAM device, electronic device, and readable storage medium provided in this application, through specific embodiments and application scenarios.

[0024] Among them, the binocular vision-inertial tightly coupled SLAM method based on neural networks can be applied to terminals, such as robot onboard computing units, and can be executed by hardware or software in the terminal.

[0025] The binocular vision-inertial tightly coupled SLAM method based on neural networks provided in this application embodiment can be executed by an electronic device or a functional module or entity within an electronic device that can implement the neural network-based binocular vision-inertial tightly coupled SLAM method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The following description uses an electronic device as the execution subject to illustrate the neural network-based binocular vision-inertial tightly coupled SLAM method provided in this application embodiment.

[0026] like Figure 1 As shown, this neural network-based binocular vision-inertial tightly coupled SLAM method includes: Step S110: Obtain the first image sequence and the second image sequence that are synchronized in time, input the first image sequence and the second image sequence into a pre-trained neural network, and obtain the first three-dimensional point map and the first descriptor corresponding to the first image sequence, as well as the second three-dimensional point map and the second descriptor corresponding to the second image sequence.

[0027] Specifically, the first image sequence can be acquired and input by the left-eye camera located on the left side of the robot, and the second image sequence can be acquired and input by the right-eye camera located on the right side of the robot. Preferably, the left-eye camera and the right-eye camera are positioned opposite each other on the robot and are on the same horizontal plane.

[0028] It is worth noting that in some embodiments, the neural network can be a Convolutional Neural Network (CNN). Further, the CNN can be a Transformer network, or any other neural network capable of outputting 3D point maps and descriptors. The advantage of using a Transformer network is the high accuracy of the output 3D point maps and descriptors, leading to better similarity matching results. The disadvantages are high computational cost and high overhead.

[0029] In some embodiments, the Transformer network can be an open-source architecture such as MASt3R, DUSt3R, or VGGT, which can output a corresponding 3D point map and its corresponding descriptor based on the input image sequence, and the output 3D point map has a realistic physical scale. Both the first and second 3D point maps are dense 3D point clouds of H*W*3.

[0030] It is worth noting that God through The 3D point map output by the network includes at least one pixel, and the god through The network can output a descriptor for each pixel, which is used to describe the physical characteristics of the pixel.

[0031] Step S120: Spatiotemporal 3D matching is performed on the first 3D point map, the first descriptor, the second 3D point map, and the second descriptor to obtain a set of matched 3D point pairs, which includes at least one set of matched 3D point pairs.

[0032] Specifically, in some embodiments, such as Figure 2 As shown, step S120 includes: S121 Based on the similarity between the first descriptor and the second descriptor, spatial matching is performed on the first three-dimensional point map and the second three-dimensional point map at the same time to obtain a spatial matching set.

[0033] S122, based on the similarity of the first descriptors at adjacent time points, perform temporal matching on the first three-dimensional point maps at adjacent time points, and / or, based on the similarity of the second descriptors at adjacent time points, perform temporal matching on the second three-dimensional point maps at adjacent time points, to obtain a temporal matching set.

[0034] S123, merge the spatial matching set and the temporal matching set to obtain the matching three-dimensional point pair set.

[0035] In this embodiment, the same pixel represented in the first 3D point map and the second 3D point map is matched by the similarity between the first descriptor and the second descriptor. The first 3D point map at adjacent time points is matched temporally by the similarity between the first descriptors at adjacent time points, or the second 3D point map at adjacent time points is matched temporally by the similarity between the second descriptors at adjacent time points. This can match the same pixel represented in the first 3D point map or the second 3D point map at different time points, and further merge the spatially matched pixel pairs and the temporally matched pixel pairs. By performing 3D information matching from both spatial and temporal dimensions, the accuracy of each pixel can be guaranteed, and the amount of computation can be reduced, providing a stable and reliable geometric constraint basis for subsequent optimization.

[0036] Further, step S121 includes: acquiring each first pixel of the first three-dimensional point map at time t. First pixel descriptor ; Obtain the second pixel points of the second 3D point map at time t. Second pixel descriptor ; Calculate the first pixel descriptor Second pixel descriptor The first similarity is used to determine the first and second pixels with the highest similarity, which are then used as the matching spatial 3D point pairs. The set of all spatial 3D point pairs is then used as the spatial matching set. This embodiment enables the matching of three-dimensional point pairs in space.

[0037] Further, step S122 includes: acquiring each first pixel of the first three-dimensional point map at time t. First pixel descriptor ; Obtain the third pixel of the first 3D point map at time t-1. The third pixel descriptor Calculate the second similarity between the first and third pixel descriptors, and use the first and third pixel points with the highest second similarity as the matched temporal 3D point pairs; use the set of each temporal 3D point pair as the temporal matching set. .

[0038] And / or, step S122 includes: acquiring each second pixel point of the second three-dimensional point map at time t. Second pixel descriptor ; Obtain the fourth pixel of each point in the second 3D point map at time t-1. The fourth pixel descriptor Calculate the third similarity between the second and fourth pixel descriptors, and use the second and fourth pixel points with the highest third similarity as the matched temporal 3D point pairs; use the set of each temporal 3D point pair as the temporal matching set. This embodiment enables the matching of three-dimensional point pairs in a temporal sequence.

[0039] Optionally, the first, second, and third similarities are all cosine similarities. All of the aforementioned descriptors are... The feature map.

[0040] In some embodiments, step S123 includes: merging spatial 3D point pairs in the spatial matching set and temporal 3D point pairs in the temporal matching set to obtain a set of matched 3D point pairs. This embodiment can achieve matching of three-dimensional point pairs in both spatial and temporal dimensions.

[0041] Through the aforementioned steps S110-S120, a binocular neural network is used to convert two time-synchronized input image sequences into dense 3D point maps and feature descriptors with realistic metric scales. Then, based on the similarity of 3D coordinates and descriptors, 3D point pair matching is performed in both the spatial (left and right eyes) and temporal (previous and subsequent frames) dimensions, constructing a scale-accurate and robust 3D data association. This transforms the original image information into scale-adjusted, structured 3D geometric information that can be directly used by the subsequent optimization framework. This fundamentally differs from the monocular neural network approach.

[0042] Step S130: Based on the matching set of three-dimensional point pairs, construct the visual constraint factor, and fuse the visual constraint factor with the IMU pre-integration factor model to obtain the tightly coupled factor model.

[0043] In some embodiments, such as Figure 3 As shown, step S130 includes: S131, calculating the geometric error for each pair of matched 3D point pairs in the set of matched 3D point pairs, where the geometric error represents the position error of each pair of matched 3D point pairs in the global coordinate system. S132, based on the geometric error, calculating the sensitivity residual corresponding to each pair of matched 3D point pairs, where the sensitivity residual represents the amount of change in geometric error caused by changes in the robot's pose. S133, calculating the sum of the position errors of all matched 3D point pairs and the sum of the sensitivity residuals of all matched 3D point pairs to obtain the total information matrix and the total residual vector representing the strength and direction of the visual constraint. S134, constructing the visual constraint factor based on the total information matrix and the total residual vector. This embodiment can realize the construction of the visual constraint factor based on the set of matched 3D point pairs M obtained in the aforementioned steps S110 and S120.

[0044] Further, in some embodiments, step S131 includes: obtaining each valid pair of matching 3D point pairs in the set of matching 3D point pairs, denoted as... ,in, Belongs to the position as The frame, Belongs to the position as For each pair of matched 3D point pairs, calculate the geometric error within the frames. Its formula is: ,in, This indicates an attitude transformation operation.

[0045] Further, in some embodiments, step S132 includes: calculating Regarding posture , Jacobian matrix and give the geometric error covariance matrix Calculate the Hessian information matrix block and residual vector contributed by the matched 3D point pair, expressed by the formula: .

[0046] Further, in some embodiments, step S133 includes: calculating the sum of the information matrix blocks and residual vectors of all matching 3D point pairs based on the Hessian information matrix blocks and residual vectors contributed by each matching 3D point pair, to obtain the total information matrix representing the strength and direction of the visual constraint. Sum of total residual vectors .

[0047] Furthermore, in some embodiments, step S134 includes: converting the total information matrix... Sum of total residual vectors Encapsulated as a visual constraint factor in quadratic form .

[0048] In some embodiments, such as Figure 3 As shown, step S130 further includes: S135, obtaining an IMU pre-integration factor model containing IMU (Inertial Measurement Unit) factors, where the IMU factors represent the key state information perceived by the robot within a certain period of time. S136, adding the visual constraint factors to the IMU pre-integration factor model and fusing them with the IMU factors to obtain a tightly coupled factor model.

[0049] This embodiment directly converts the matched 3D point pairs into Hessian information matrices and uses this to construct visual constraint factors. This mechanism is key to achieving tight coupling with the IMU. It completely avoids the indirect process of first estimating the relative pose and realizes the shortest path fusion of visual geometric information and inertial measurement information at the optimization level, thereby improving accuracy and robustness.

[0050] The key state information includes the robot's pose, velocity, rotation angle, and IMU bias. Visual constraint factors can impose directional and intensity constraints on the IMU factors in the IMU pre-integration factor model, thereby optimizing the robot's state estimation information and dense 3D point cloud map in subsequent solutions.

[0051] Step S140: Use a nonlinear optimizer to solve the tightly coupled factor model to obtain the robot's state estimation information and / or dense 3D point cloud map.

[0052] Specifically, step S140 includes: adjusting the visual constraint factor and IMU factor in the tightly coupled factor model using a nonlinear optimizer, and using the robot state information and pose information when the visual constraint factor and IMU factor are maximized as state estimation information and dense 3D point cloud map. The state estimation information may include information such as the robot's pose, velocity, and deviation.

[0053] Specifically, the pose information when the visual constraint factor and IMU factor are maximized is used as a dense 3D point cloud map. This involves transforming the 3D point map corresponding to the pose information in the state estimation information to a unified world coordinate system and fusing them to generate a real-time, metric-scaled dense 3D environment map.

[0054] The binocular vision-inertial tightly coupled SLAM method based on neural networks provided in this application directly regresses a 3D point map with a metric scale through neural networks, and comprehensively utilizes 3D information in both spatial and temporal dimensions for matching to obtain at least one pair of matched 3D point points. This provides a stable, accurate, and scaled geometric constraint foundation for subsequent optimization. Furthermore, the Hessian information matrix and residual vector of the matched 3D point pairs with respect to the system state variables are constructed using the matching 3D point pairs and encapsulated as a new visual constraint factor. This visual constraint factor and IMU factor are integrated into a unified, metric-scale tightly coupled factor graph model, and joint state estimation is performed through nonlinear optimization. This completely avoids the indirect process of first estimating relative pose, and realizes the fusion of visual geometric information and inertial measurement information at the optimization level, improving accuracy and robustness, and achieving high-precision, high-fidelity, and highly robust real-time positioning and map construction.

[0055] In some embodiments, step S136 further includes: when the computational cost of the visual constraint factor added to the IMU pre-integration factor model exceeds a preset computational cost, refining the visual constraint factor and IMU factor in the tightly coupled factor model into marginalized prior factors and retaining them in the tightly coupled factor model.

[0056] This embodiment not only ensures that historical information is not discarded, but also controls the amount of computation.

[0057] To better understand the technical solution of this application, the following is in conjunction with the appendix. Figure 4 The technical solution of this application is further described below: S101 receives the input binocular image sequence, namely the left eye image sequence and the right eye image sequence.

[0058] S102, input the stereo image sequence into the Transformer network, such as MASt3R, DUSt3R, VGGT, etc.

[0059] S013, the Transformer network parses the stereo image sequence and outputs a 3D point map with a metric scale and corresponding descriptors. The 3D point map includes the left-eye 3D point map corresponding to the left-eye image sequence and the right-eye 3D point map corresponding to the right-eye image sequence. The descriptors include the left-eye descriptor of the left-eye 3D point map and the right-eye descriptor of the right-eye 3D point map.

[0060] S104, perform spatial matching of the left-eye 3D point map and the right-eye 3D point map at the same time point based on the left-eye descriptor and the right-eye descriptor, and perform temporal matching of the left-eye 3D point map or the right-eye 3D point map at previous and subsequent time points based on the left-eye descriptor or the right-eye descriptor, to obtain a set of matched 3D point pairs including at least one pair of matched 3D point pairs.

[0061] S105, constructing Hession visual constraint factors based on matching sets of 3D point pairs.

[0062] It should be noted that the aforementioned steps S101 to S105 are performed on the robot's GPU.

[0063] S106. While executing steps S101 to S105, or after steps S101 to S105, acquire IMU data, including the robot's pose, speed, IMU deviation, etc.

[0064] S107, Based on IMU data, construct an IMU pre-integration factor model.

[0065] S108, the visual constraint factor obtained in step S105 is fused with the IMU factor of the IMU pre-integration factor model in step S107 to obtain the tightly coupled factor model. The IMU factor is obtained from IMU data.

[0066] S109. A nonlinear optimizer is used to optimize the IMU factor and visual constraint factor in the tightly coupled factor model, so that the constraints represented by the visual constraint factor and IMU factor are satisfied to the greatest extent. The solution is the optimal state estimation information such as robot pose, velocity, and deviation. At the same time, using the optimal robot pose, the corresponding 3D point map is transformed to a unified world coordinate system to obtain a dense 3D point cloud map.

[0067] The method described above in the embodiments of this application can achieve high-precision, high-fidelity, and robust real-time positioning and map construction.

[0068] The binocular vision-inertial tightly coupled SLAM method based on neural networks provided in this application can be executed by a binocular vision-inertial tightly coupled SLAM device based on neural networks. This application uses the execution of the method by a binocular vision-inertial tightly coupled SLAM device based on neural networks as an example to illustrate the binocular vision-inertial tightly coupled SLAM device based on neural networks provided in this application.

[0069] This application also provides a binocular vision-inertial tightly coupled SLAM device based on neural networks, such as... Figure 5As shown, the binocular vision-inertial tightly coupled SLAM device based on a neural network includes: an acquisition module 110, used to acquire a time-synchronized first image sequence and a second image sequence, inputting the first image sequence and the second image sequence into a pre-trained neural network to obtain a first 3D point map and a first descriptor corresponding to the first image sequence, and a second 3D point map and a second descriptor corresponding to the second image sequence; a matching module 120, used to perform spatiotemporal 3D matching of the first 3D point map, the first descriptor, the second 3D point map, and the second descriptor to obtain a set of matched 3D point pairs, the set of matched 3D point pairs including at least one pair of matched 3D point pairs; a fusion module 130, used to construct a visual constraint factor based on the set of matched 3D point pairs, and fuse the visual constraint factor with an IMU pre-integration factor model to obtain a tightly coupled factor model; and a solution module 140, used to solve the tightly coupled factor model using a nonlinear optimizer to obtain robot state estimation information and / or a dense 3D point cloud map.

[0070] According to the neural network-based binocular vision-inertial tightly coupled SLAM device provided in this application embodiment, a 3D point map with a metric scale is directly regressed through the neural network. It then comprehensively utilizes 3D information from both spatial and temporal dimensions for matching, obtaining at least one pair of matched 3D point pairs. This provides a stable, accurate, and scaled geometric constraint basis for subsequent optimization. Furthermore, the Hessian information matrix and residual vector of the matched 3D point pairs with respect to the system state variables are constructed and encapsulated as a new visual constraint factor. This visual constraint factor and IMU factor are integrated into a unified, metric-scale tightly coupled factor graph model. Joint state estimation is performed through nonlinear optimization, completely avoiding the indirect process of first estimating relative pose. This achieves the shortest path fusion of visual geometric information and inertial measurement information at the optimization level, improving accuracy and robustness, and achieving high-precision, high-fidelity, and highly robust real-time positioning and map construction.

[0071] In some embodiments, the matching module 120 is further configured to: spatially match the first three-dimensional point map and the second three-dimensional point map at the same time based on the similarity between the first descriptor and the second descriptor to obtain a spatial matching set; temporally match the first three-dimensional point maps at adjacent time based on the similarity between the first descriptors at adjacent time, and / or temporally match the second three-dimensional point maps at adjacent time based on the similarity between the second descriptors at adjacent time to obtain a temporal matching set; and merge the spatial matching set and the temporal matching set to obtain the matching three-dimensional point pair set.

[0072] In some embodiments, the matching module 120 is further configured to obtain a first pixel descriptor for each first pixel point of the first three-dimensional point map at time t; obtain a second pixel descriptor for each second pixel point of the second three-dimensional point map at time t; calculate a first similarity between the first pixel descriptor and the second pixel descriptor; take the first pixel point and the second pixel point with the highest first similarity as a matched spatial three-dimensional point pair; and take the set of each spatial three-dimensional point pair as the spatial matching set.

[0073] In some embodiments, the matching module 120 is further configured to obtain the first pixel descriptor of each first pixel point of the first three-dimensional point map at time t; obtain the third pixel descriptor of each third pixel point of the first three-dimensional point map at time t-1; calculate the second similarity between the first pixel descriptor and the third pixel descriptor; take the first pixel point and the third pixel point with the highest second similarity as the matched temporal three-dimensional point pair; and take the set of each temporal three-dimensional point pair as the temporal matching set.

[0074] In some embodiments, the matching module 120 is further configured to obtain the second pixel descriptor of each second pixel point in the second three-dimensional point map at time t; obtain the fourth pixel descriptor of each fourth pixel point in the second three-dimensional point map at time t-1; calculate the third similarity between the second pixel descriptor and the fourth pixel descriptor; take the second pixel point and the fourth pixel point with the highest third similarity as the matched temporal three-dimensional point pair; and take the set of each temporal three-dimensional point pair as the temporal matching set.

[0075] In some embodiments, the matching module 120 is further configured to merge the spatial three-dimensional point pairs in the spatial matching set and the temporal three-dimensional point pairs in the temporal matching set to obtain the matching three-dimensional point pair set.

[0076] In some embodiments, the fusion module 130 is further configured to: calculate a geometric error for each pair of matched 3D point pairs in the set of matched 3D point pairs, wherein the geometric error represents the position error of each pair of matched 3D point pairs in the global coordinate system; calculate a sensitivity residual corresponding to each pair of matched 3D point pairs based on the geometric error, wherein the sensitivity residual represents the amount of change in the geometric error caused by the robot's pose change; calculate the sum of the position errors of all matched 3D point pairs and the sum of the sensitivity residuals of all matched 3D point pairs to obtain a total information matrix and a total residual vector representing the strength and direction of the visual constraint; and construct the visual constraint factor based on the total information matrix and the total residual vector.

[0077] In some embodiments, the fusion module 130 is further configured to obtain an IMU pre-integration factor model containing IMU factors, wherein the IMU factors represent key state information perceived by the robot within a certain period of time, including the robot's pose, velocity, rotation angle, and IMU deviation; the visual constraint factors are added to the IMU pre-integration factor model and fused with the IMU factors to obtain the tightly coupled factor model.

[0078] In some embodiments, the fusion module 130 is further configured to extract the visual constraint factors and the IMU factors in the tightly coupled factor model into marginalized prior factors and retain them in the tightly coupled factor model when the computational cost of the visual constraint factors added to the IMU pre-integration factor model exceeds a preset computational cost.

[0079] In some embodiments, the solver module 140 is further configured to use the nonlinear optimizer to adjust the visual constraint factor and the IMU factor in the tightly coupled factor model, and use the robot state information and pose information when the visual constraint factor and the IMU factor are maximized as the state estimation information and the dense three-dimensional point cloud map.

[0080] The binocular vision-inertial tightly coupled SLAM device based on neural networks in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific devices.

[0081] The binocular vision-inertial tightly coupled SLAM device based on neural networks in this application embodiment can be a device with an operating system. This operating system can be Microsoft (Windows), Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0082] The binocular vision-inertial tightly coupled SLAM device based on neural networks provided in this application embodiment can achieve... Figures 1 to 4 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0083] In some embodiments, such as Figure 6 As shown, this application embodiment also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, it implements the various processes of the above-described embodiment of the binocular vision-inertial tightly coupled SLAM method based on neural networks and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0084] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0085] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described embodiment of the binocular vision-inertial tightly coupled SLAM method based on neural networks and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0086] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0087] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described neural network-based binocular vision-inertial tightly coupled SLAM method.

[0088] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0089] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiment of the binocular vision-inertial tightly coupled SLAM method based on neural networks, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0090] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0091] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0093] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0094] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0095] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A neural network based binocular vision inertial tight coupling simultaneous localization and mapping (SLAM) method, characterized in that, include: A first image sequence and a second image sequence synchronized with time are acquired. The first image sequence and the second image sequence are input into a pre-trained neural network to obtain a first three-dimensional point map and a first descriptor corresponding to the first image sequence, and a second three-dimensional point map and a second descriptor corresponding to the second image sequence. Spatiotemporal 3D matching is performed on the first 3D point map, the first descriptor, the second 3D point map, and the second descriptor to obtain a set of matched 3D point pairs, wherein the set of matched 3D point pairs includes at least one pair of matched 3D point pairs. Based on the set of matched 3D point pairs, a visual constraint factor is constructed, and the visual constraint factor is fused with the IMU pre-integration factor model to obtain a tightly coupled factor model. The tight coupling factor model is solved using a nonlinear optimizer to obtain the robot's state estimation information and / or a dense 3D point cloud map.

2. The method of claim 1, wherein, The step of performing spatiotemporal 3D matching on the first 3D point map, the first descriptor map, the second 3D point map, and the second descriptor map to obtain a set of matched 3D point pairs includes: Based on the similarity between the first descriptor and the second descriptor, the first 3D point map and the second 3D point map at the same time are spatially matched to obtain a spatial matching set; Based on the similarity of the first descriptors at adjacent time points, temporal matching is performed on the first three-dimensional point maps at adjacent time points, and / or, based on the similarity of the second descriptors at adjacent time points, temporal matching is performed on the second three-dimensional point maps at adjacent time points, to obtain a temporal matching set; The spatial matching set and the temporal matching set are merged to obtain the matching three-dimensional point pair set.

3. The method of claim 2, wherein, Based on the similarity between the first descriptor and the second descriptor, spatial matching is performed on the first 3D point map and the second 3D point map at the same time to obtain a spatial matching set, including: Obtain the first pixel descriptor of each first pixel point in the first 3D point map at time t; Obtain the second pixel descriptor for each second pixel point of the second 3D point map at time t; Calculate the first similarity between the first pixel descriptor and the second pixel descriptor, and take the first pixel point and the second pixel point with the highest first similarity as a matching spatial three-dimensional point pair; The set of each of the aforementioned three-dimensional point pairs is taken as the spatial matching set.

4. The method of claim 3, wherein, The first three-dimensional point maps at adjacent time points are temporally matched based on the similarity of the first descriptors at adjacent time points, and / or, the second three-dimensional point maps at adjacent time points are temporally matched based on the similarity of the second descriptors at adjacent time points, to obtain a temporally matched set, including: Obtain the first pixel descriptor of each first pixel point in the first 3D point map at time t; Obtain the third pixel descriptor of each third pixel point in the first 3D point map at time t-1; Calculate the second similarity between the first pixel descriptor and the third pixel descriptor, and take the first pixel point and the third pixel point with the highest second similarity as the matched temporal three-dimensional point pair; The set of each of the aforementioned temporal 3D point pairs is used as the temporal matching set; and / or Obtain the second pixel descriptor for each second pixel point of the second 3D point map at time t; Obtain the fourth pixel descriptor of each fourth pixel point in the second three-dimensional point map at time t-1; Calculate the third similarity between the second pixel descriptor and the fourth pixel descriptor, and take the second pixel point and the fourth pixel point with the highest third similarity as the matched temporal three-dimensional point pair; The set of each of the aforementioned time-series three-dimensional point pairs is used as the time-series matching set.

5. The method of claim 4, wherein, The step of merging the spatial matching set and the temporal matching set to obtain the matching three-dimensional point pair set includes: The spatial 3D point pairs in the spatial matching set and the temporal 3D point pairs in the temporal matching set are merged to obtain the matching 3D point pair set.

6. The method according to any one of claims 1 to 5, characterized in that, The construction of visual constraint factors based on the set of matched 3D point pairs includes: For each pair of matching 3D point pairs in the set of matching 3D point pairs, calculate the geometric error, where the geometric error represents the positional error of each pair of matching 3D point pairs in the global coordinate system. Based on the geometric error, the sensitivity residual corresponding to each pair of matched 3D points is calculated, and the sensitivity residual represents the amount of change in the geometric error caused by the change in the robot's pose. The sum of positional errors of all the matched 3D point pairs and the sum of sensitivity residuals of all the matched 3D point pairs are calculated to obtain the total information matrix and the total residual vector representing the visual constraint strength and direction. The visual constraint factor is constructed based on the total information matrix and the total residual vector.

7. The method of claim 6, wherein, The process of fusing the visual constraint factor with the IMU pre-integration factor model to obtain a tightly coupled factor model includes: Obtain an IMU pre-integration factor model containing IMU factors, wherein the IMU factors represent the key state information perceived by the robot within a certain period of time, and the key state information includes the robot's pose, velocity, rotation angle, and IMU bias. The visual constraint factor is added to the IMU pre-integration factor model and fused with the IMU factor to obtain the tightly coupled factor model.

8. The method of claim 7, wherein, When the computational cost of the visual constraint factor added to the IMU pre-integration factor model exceeds the preset computational cost, the visual constraint factor and the IMU factor in the tightly coupled factor model are extracted into marginalized prior factors and retained in the tightly coupled factor model.

9. The method of claim 7, wherein, The process of solving the tightly coupled factor model using a nonlinear optimizer to obtain robot state estimation information and / or a dense 3D point cloud map includes: The nonlinear optimizer is used to adjust the visual constraint factor and the IMU factor in the tightly coupled factor model. The robot state information and pose information when the visual constraint factor and the IMU factor are maximized are used as the state estimation information and the dense 3D point cloud map. 10.A binocular vision inertial tight-coupling SLAM device of neural network, characterized in that, include: The acquisition module is used to acquire a time-synchronized first image sequence and a second image sequence, input the first image sequence and the second image sequence into a pre-trained neural network, and obtain a first three-dimensional point map and a first descriptor corresponding to the first image sequence, as well as a second three-dimensional point map and a second descriptor corresponding to the second image sequence. The matching module is used to perform spatiotemporal three-dimensional matching of the first three-dimensional point map, the first descriptor, the second three-dimensional point map, and the second descriptor to obtain a set of matched three-dimensional point pairs, wherein the set of matched three-dimensional point pairs includes at least one pair of matched three-dimensional point pairs. The fusion module is used to construct a visual constraint factor based on the set of matched 3D point pairs, and fuse the visual constraint factor with the IMU pre-integration factor model to obtain a tightly coupled factor model; The solver module is used to solve the tightly coupled factor model using a nonlinear optimizer to obtain the robot's state estimation information and / or a dense 3D point cloud map.

Images