Joint training network device

Abstract

A joint training network includes a multi-head module, which includes: a network input; a feature network, which is coupled to the network input and includes a feature detector decoder that outputs an interest point and a descriptor generator decoder that outputs a descriptor, the feature detector decoder and the descriptor generator decoder being coupled in parallel; a depth network, which includes a monocular depth prediction decoder and outputs a depth map; a flow network, which includes an image segmentation decoder and outputs a segmented image; a segmentation network, which includes a warp module that outputs a rotation and translation and an input warp signal to a segmentation decoder that outputs a residual stream; and a pose network, which includes a fully connected pose estimator, which is coupled to an adder that receives an input from the pose estimator and a residual stream from the segmentation decoder, and the adder outputs an optical flow.

Description

Technical Field

[0001] The present invention relates to autonomous driving, and visual positioning and map creation based on deep learning, and more particularly to a joint training network device. Background Art

[0002] Current Visual Simultaneous Localization and Mapping (VSLAM) methods are hampered by the ill-posed problem of 3D reconstruction from 2D images. Summary of the Invention

[0003] Based on this, it is necessary to provide a joint training network device for the ill-posed problem of 3D reconstruction from 2D images.

[0004] An example joint training network apparatus includes a multi-head module comprising: a network input; a feature network coupled to the network input and comprising a feature detector decoder that outputs an interest point and a descriptor generator decoder that outputs a descriptor, the feature detector decoder and the descriptor generator decoder being coupled in parallel; a depth network coupled to the network input and comprising a monocular depth prediction decoder that outputs a depth map; a flow network coupled to the network input and comprising an image segmentation decoder that outputs a segmented image; a segmentation network coupled to the network input and comprising a warp module that outputs a rotation and translation and an input warp signal to a segmentation decoder that outputs a residual stream; and a pose network coupled to the network input and comprising a fully connected pose estimator coupled to an adder that receives an input from the pose estimator and a residual stream from the segmentation decoder, the adder outputting an optical flow.

[0005] The joint training network arrangement overcomes the problem that current visual simultaneous localization and mapping methods are hampered by the ill-posed problem of 3D reconstruction from 2D images. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In the attached figure:

[0007] Figure 1 is a first exemplary system diagram according to one embodiment of the present invention;

[0008] Figure 2 is a second exemplary system diagram according to one embodiment of the present invention;

[0009] Figure 3 is an example joint training framework for learning pose, features, depth, flow, and segmentation according to one embodiment of the present invention;

[0010] Figure 4An example deep learning visual odometry or visual simultaneous localization and mapping platform with long term short term memory according to one embodiment of the present invention;

[0011] Figure 5 is an example keyframe classifier model according to one embodiment of the present invention;

[0012] Figure 6 is an example attention-based model for loop closure according to one embodiment of the present invention;

[0013] Figure 7 is an example joint training framework for learning pose, features, depth, flow, and segmentation with long short-term memory according to one embodiment of the present invention; and

[0014] Figure 8 is an example multi-head module architecture according to one embodiment of the present invention. DETAILED DESCRIPTION

[0015] The embodiments listed below are written only to illustrate the application of the device and method, rather than to limit the scope. Modifications of equivalent forms of the device and method should be classified as within the scope of the claims.

[0016] Certain terms are used in the following description and claims to refer to specific system components. As will be appreciated by those skilled in the art, different companies may refer to components and / or methods by different names. This document is not intended to distinguish between components and / or methods that have the same function but differ in name.

[0017] In the following discussion and in the claims, the terms "including" and "comprising" are used in an open-ended fashion, and thus can be interpreted to mean "including, but not limited to... ." Likewise, the term "coupled" is intended to mean either an indirect or direct connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

[0018] Figure 1 An example hybrid computing system 100 is described that can be used to implement a neural network associated with the operation of one or more parts or steps of the process. In this example, the processors associated with the hybrid system include a field programmable gate array (FPGA) 122, a graphics processor unit (GPU) 120, and a central processing unit (CPU) 118.

[0019] CPU 118, GPU 120, and FPGA 122 have the ability to provide neural networks. The CPU is a general-purpose processor that can perform many different functions. Its versatility results in the ability to perform multiple different tasks. However, the CPU is limited in its ability to process multiple data streams, and its functionality with respect to neural networks is limited. The GPU is a graphics processor with many small processing cores that can sequentially process parallel tasks. The FPGA is a field-programmable device that has the ability to be reconfigured and hard-wired to perform any function that can be programmed into a CPU or GPU. Because FPGAs are programmed in circuit form, they are many times faster than CPUs and significantly faster than GPUs.

[0020] There are other types of processors that a system may include, such as accelerated processing units (APUs) that include CPUs with GPU elements on a chip, and digital signal processors (DSPs) that are designed to perform high-speed digital data processing. Application-specific integrated circuits (ASICs) can also perform the hardwired functions of an FPGA; however, the lead time for designing and producing ASICs is on the order of several quarters of a year, rather than the quick turnaround available in programming FPGAs.

[0021] Graphics processor unit 120, central processing unit 118, and field programmable gate array 122 are connected and connected to memory interface controller 112. The FPGA is connected to the memory interface through programmable logic circuits and then to memory interconnect 130. Because FPGAs operate at very high bandwidths, this additional device is used to minimize the circuitry required to perform memory tasks from the FPGA. Memory and interface controller 112 is also connected to permanent storage disk 110, system memory 114, and read-only memory (ROM) 116.

[0022] Figure 1 The system can be used to program and train the FPGA. The GPU works well with unstructured data and can be used for training. Once the data has been trained, a deterministic inference model can be found, and the CPU can use the model data determined by the GPU to program the FPGA.

[0023] The memory interfaces and controllers are connected to a central interconnect 124, which is further connected to GPU 120, CPU 118, and FPGA 122. Central interconnect 124 is further connected to input and output interfaces 128 and network interface 126.

[0024] Figure 2A second example hybrid computing system 200 is described that may be used to implement a neural network associated with the operations of one or more portions or steps of process 1000. In this example, processors associated with the hybrid system include a field programmable gate array (FPGA) 210 and a central processing unit (CPU) 220.

[0025] The FPGA is electrically connected to an FPGA controller 212, which interfaces with a direct memory access (DMA) 218 interface. The DMA is connected to an input buffer 214 and an output buffer 216, which are coupled to the FPGA to buffer data entering and leaving the FPGA, respectively. The DMA 218 includes two first-in-first-out (FIFO) buffers, one for the host CPU and the other for the FPGA, allowing data to be written to and read from the appropriate buffers.

[0026] On the CPU side of the DMA is a main switch 228, which shuttles data and commands to the DMA. The DMA is also connected to a synchronous dynamic random access memory (SDRAM) controller 224, which allows data to be transferred to and from the FPGA to the CPU 220. The SDRAM controller is also connected to an external SDRAM 226 and the CPU 220. The main switch 228 is connected to a peripheral interface 230. A flash memory controller 222 controls permanent storage and is connected to the CPU 220.

[0027] Possible solutions for deep visual odometry (VO) and visual simultaneous localization and mapping (VSLAM) can be constructed using a convolutional neural network (CNN)-long short-term memory (LSTM) framework. In one example solution, a CNN architecture combining a deep network, a flow network, a segmentation network, and a feature network can be utilized. Within the LSTM architecture, a two-layer LSTM with an attention mechanism and multi-stage training is designed to simulate keyframe selection, local / global pose graph optimization, and loop closure. The attention mechanism selectively focuses on relevant terms while ignoring others, allowing the decoder to decide which parts of the source sentence to pay attention to.

[0028] Feature extraction, disparity / depth estimation, optical flow estimation, and pose estimation are currently utilized in visual odometry (VO) or visual simultaneous localization and mapping (VSLAM) frameworks. VO or VSLAM methods are categorized as direct and indirect, where the former estimates pose directly from images, while the latter extract features and compute pose from 2D feature matching or 2D feature-to-3D reconstruction feature matching. VO or VSLAM platforms currently consist of a front-end and a back-end, where the front-end determines the pose and adds the newly reconstructed scene to the map, while the back-end optimizes the set of poses as well as the map in a nonlinear optimization framework. Keyframe structures can be applied to VO or VSLAM to improve the efficiency of both the front-end and back-end.

[0029] An example solution provides a pose network that is jointly trained with an optical flow network, a deep network, a segmentation network, and a feature network for visual odometry and simultaneous visual localization and mapping.

[0030] In an example solution, the framework of multi-task learning in deep VO or VLSAM involves feature network, deep network, flow network, semantic segmentation network and pose network, as shown in Figure 3 shown.

[0031] Figure 3 An example joint training network apparatus is described, comprising at least two joint training encoders 316, 318 that receive a plurality of frames 310, 312, 314 and output a network input. A feature network 324 is coupled to the network input and comprises a feature detector decoder 332 that outputs interest points 334 and a descriptor generator decoder 328 that outputs descriptors 330, the feature detector decoder 332 and the descriptor generator decoder 328 being coupled in parallel. A depth network is coupled to the network input and comprises a monocular depth prediction decoder 320 that outputs a depth map 322. A stream network is coupled to the network input and comprises an image segmentation decoder 336 that outputs a segmented image 340. A segmentation network 344 is coupled to the network input and comprises a warp module 342 that rotates and translates the segmentation decoder output 354 and inputs a warp signal to output a residual stream 346. The pose network is coupled to the network input and includes a fully connected pose estimator 348, which is coupled to an adder 350 that receives input from the pose estimator and the residual stream 346 from the segmentation decoder, and outputs an optical flow 352. Monocular depth prediction can be performed by DepthNet, image segmentation can be performed by SegmentNet, the pose estimator can be performed by PoseNet, and the flow network can be FlowNet.

[0032] A feature is a single, measurable characteristic of an object in an image. A feature vector is an n-dimensional vector that represents the numerical characteristics of that object. The feature network is a convolutional neural network that identifies the features of an object and describes them as an n-dimensional vector. The feature network can include a feature detector decoder that outputs interest points and a descriptor generator decoder that outputs descriptors.

[0033] DepthNet is a convolutional long short-term memory (LSTM) network architecture for depth prediction from monocular video sequences. A LSTM recurrent neural network (RNN) learns sequential reasoning and predicts a depth map for the current image frame based on the appearance of objects in the current frame and temporally proximal frames. The deep network can include a monocular depth prediction decoder.

[0034] Optical flow estimation utilizes per-pixel localization and finds correspondences between two input images. This involves learning image feature representations and matching them at different locations in the two images. FlowNet is a convolutional neural network that predicts optical flow directly from two input images. In FlowNet, information is spatially compressed and then refined. Flow networks can include image segmentation decoders.

[0035] SegmentNet is a convolutional neural network that uses convolution and deconvolution to produce segmented images. SegmentNet uses convolutional neural networks to remove background information for semantic segmentation and create object mask images. The semantic segmentation network can include a warp module that outputs rotation and translation as well as input warp signals to the semantic decoder that outputs the residual stream.

[0036] PoseNet is a convolutional neural network that regresses a six-degree-of-freedom camera pose from a single image. PoseNet uses structure from motion to label data to generate a regression dataset for camera pose, and transfer learning to train a pose regressor that is pre-trained as a classifier based on an image recognition dataset. The pose network may include a fully connected pose estimator coupled to an adder that receives input from the pose estimator and a residual stream from a semantic decoder.

[0037] exist Figure 3 In

[15] , the input is a monocular image from a camera, where the pose network and flow network take two images as input, and the other networks take one image as input. The networks share an encoder to extract hierarchical feature maps, and the decoder or regression layers are coupled to different outputs.

[0038] The feature network is designed for both feature detection and description generation. Two different decoders are shown, one for feature detection and localization, and one for description generation. In other example networks, a single decoder can be used for both tasks. Feature matching is performed using similarity between feature descriptors (e.g., k-NN). Matched points can be used in the loss function of the pose network.

[0039] The segmentation network can generate a pixel-by-pixel scene parsing, where it is recommended to remove some objects (such as sky, water-like objects, trees, mirrors, vehicles, pedestrians, etc.) from the loss function of the pose network. Road surfaces and static objects in streets or highways can be removed from the loss function of the residual flow estimation. Similarly, the extracted features can be classified based on semantic segmentation and the extracted features contribute to the loss function accordingly.

[0040] The depth network can infer pixel depth from a single image instead of stereo images. Depth consistency can be defined in a loss function with contributions from segmentation and optical flow. The optical flow network can estimate residual flow based on the camera motion estimated from the pose network. Depth can contain structural constraints for loss determination in the flow network and the pose network. Optical flow and pose estimates can be cross-checked to include consistency constraints (such as forward and backward warping).

[0041] The pose network in this joint training platform generates the output of VO or VSLAM (i.e., rotation R and translation t).

[0042] The unsupervised learning loss function can be defined to include an image appearance loss term based on view synthesis through depth, camera pose and residual flow, a smoothness loss term based on edge-aware image and geometry-aware depth smoothness, and a consistency loss term for camera pose, which can include rotation and translation, segmentation, residual optical flow, etc.

[0043] Segmentation labels can be implicitly applied as weights or masks in the guided loss function, and features can be applied together with segmentation masks or segmentation weights to the part of the loss that includes pose or flow contributions.

[0044] The keyframe pool can be constructed by adding or removing keyframes to avoid redundancy.

[0045] Figure 4 We describe a recurrent neural network with long short-term memory and an attention-based deep VO / VSLAM system. Figure 4An example two-layer long short-term memory (LSTM) for determining camera pose is described. The example includes at least two jointly trained encoders 416, 418 that receive a plurality of frames 410, 412, 414 and output network inputs. A first set of LSTMs 420-426 is connected to the outputs of the jointly trained encoders. A second set of LSTMs 430-436 is connected to the outputs of the first set of LSTMs 420-426. The outputs of the second set of LSTMs 430-436 are connected to a fully connected pose network 440 and output a camera pose 442. The first LSTM can learn keyframe classification selection based on pose transitions and windowed profile guided optimization, and the second LSTM can learn pose dynamics and refine the global pose based on profile guided optimization with loop closure.

[0046] Figure 4 Describes an example two-layer LSTM network for estimating camera pose based on input from a convolutional neural network encoder. The first LSTM layer can perform learning of short-term pose transitions for keyframe selection and windowed contour-guided optimization, while the second layer can perform learning of long-term pose dynamics and refinement of the global pose using contour-guided optimization with loop closure.

[0047] Figure 5 An example keyframe classifier module is described, including a first keyframe encoder 512 having a first keyframe encoder input that receives a plurality of frames 510 and a first keyframe encoder output. A second keyframe encoder 514 includes a second keyframe encoder input coupled to the first keyframe encoder input and receiving the plurality of frames. The second keyframe encoder output is coupled in parallel with the first keyframe encoder output and is coupled to a fully connected classifier layer 516 that outputs a classified output. A keyframe selection switch 518 is coupled to the first keyframe encoder output and the classified output, wherein the switch classifies a possible keyframe 520 from the plurality of frames as a keyframe if the difference between the previous keyframe and the possible keyframe is greater than a predetermined threshold. A keyframe detection module receives the first keyframe encoder output and selects a keyframe based on the difference between the first keyframe encoder output and at least one of the first feature output and the second feature output.

[0048] The output of the first layer can be filtered to remove redundancy by a keyframe detection model, which is trained to select keyframes in a switch-like manner based on the output of the encoder, e.g. Figure 5As shown. Rotation and translation can be discarded in the pose network for keyframe estimation, instead measuring the difference in feature maps from the encoder. Initially, the current frame can be compared with the previous frame; if the difference is greater than a predetermined delta from the reference frame, the keyframe can be transferred to the current frame; if not, the reference frame can remain unchanged and the current frame can be indexed to the next frame.

[0049] Figure 6 An example loop closure module is depicted, comprising a first loop closure encoder 614 that receives an input 610 and is coupled to a first feature detector decoder 624 that outputs a first interest point and a first descriptor generator decoder 622 that outputs a first descriptor. The first feature detector decoder and the first descriptor generator decoder are coupled in parallel and output a first feature output. A second loop closure encoder 616 receives an input 612 and is coupled to a second feature detector decoder 628 that outputs a second interest point and a second descriptor generator decoder 626 that outputs a second descriptor. The second feature detector decoder and the second descriptor generator decoder are coupled in parallel and output a second feature output. A fully connected similarity layer 680 receives the first feature output and the second feature output and outputs a similarity 682.

[0050] The second LSTM layer can be designed to simulate global pose optimization using a loop closure mechanism. The loop closure mechanism model can be simulated by a self-attention mechanism similar to the Transformer, where the similarity is measured by the feature network output, such as Figure 6 shown.

[0051] The global / absolute pose estimate can be calculated in the second LSTM layer by accumulating the predicted relative poses from the first LSTM layer.

[0052] The loop closure is defined by similarity in the attention mechanism to measure whether loop closure is achieved. Features can be explicitly defined via the feature network. Attention can be based on a spatial-temporal attention mechanism instead of the current time-based attention mechanism.

[0053] Figure 7 An example joint training framework is described that includes at least two joint training encoders 716, 718 that receive a plurality of frames 710, 712, 714 and output network inputs. A first long short-term memory (LSTM) 720 is coupled in parallel to a first multi-head module 722 and a keyframe classifier module 724. A second long short-term memory 726 is coupled to the output of the keyframe classifier module 724 and in parallel to a second multi-head module 728 and a loop closure module 780. The second long short-term memory 726 estimates a global pose by accumulating a set of predicted related poses from the first long short-term memory 720.

[0054] The encoder-LSTM network can be trained simultaneously with the flow network, segmentation network, feature network, deep network, two-layer LSTM, keyframe classifier and attention-based loop closure mechanism, such as Figure 7 shown.

[0055] Multi-head module Figure 8 As shown in Figure 1 The training process can be performed in multiple stages. The first stage can be for Figure 1 The first stage can be processed by the CNN encoder framework shown in the figure. The second stage can be processed by the CNN encoder in the first layer LSTM framework. The third stage can be processed by the CNN encoder in the two-layer LSTM framework.

[0056] Figure 8 A multi-head module is depicted, including a network input 810. A feature network 824 is coupled to the network input 810 and includes a feature detector decoder 832 that outputs interest points 834 and a descriptor generator decoder 828 that outputs descriptors 830. The feature detector decoder 832 and the descriptor generator decoder 828 are coupled in parallel. A depth network is coupled to the network input 810 and includes a monocular depth prediction decoder 820 that outputs a depth map 822. A flow network is coupled to the network input 810 and includes an image segmentation decoder 836 that outputs a segmented image 840. A segmentation network 844 is coupled to the network input 810 and includes a warp module 842 that outputs 854 rotations and translations and a warp signal to the segmentation decoder that outputs a residual stream 846. A pose network is coupled to the network input 810 and includes a fully connected pose estimator 848 that is coupled to an adder 850 that receives input from the pose estimator and the residual stream 846 from the segmentation decoder. The adder outputs an optical flow 852. Monocular depth prediction can be performed by DepthNet, image segmentation can be performed by SegmentNet, pose estimator can be performed by PoseNet, and the flow network can be FlowNet.

[0057] The first stage may be processed to train the at least two jointly trained encoders, the second stage may be processed to train the at least two jointly trained encoders, the first long short-term memory, the multi-head module, and the keyframe classifier module. The third stage may be processed to train the at least two jointly trained encoders, the second long short-term memory, the second stage multi-head module, the second stage keyframe classifier module, and the loop closure module.

[0058] In one example of the proposed CNN architecture, the deep network, flow network, segmentation network, and feature network are jointly trained. In the LSTM architecture, a two-layer LSTM with attention mechanism and multi-stage training performs keyframe selection, local / global pose graph optimization, and loop closure.

[0059] Those skilled in the art will appreciate that the various illustrative blocks, modules, elements, components, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative blocks, modules, elements, components, methods and algorithms have been generally described above in terms of their functions. Whether this function is implemented as hardware or software depends on the specific application and the design constraints imposed on the system. Technicians can implement the described functions in different ways for each specific application. Various components and blocks can be arranged differently (e.g., arranged in different orders, or divided in different ways), without departing from the scope of this subject technology.

[0060] It should be understood that the specific order or hierarchy of steps in the disclosed processes is an illustration of example approaches. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the processes can be rearranged. Some steps can be performed simultaneously. The accompanying method claims present elements of the various steps in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0061] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. The previous description provides various examples of the subject technology, and the subject technology is not limited to these examples. Various modifications to these aspects may be obvious to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are to be given the full scope consistent with the language claims, wherein, unless otherwise stated, reference to an element in the singular is not intended to mean "one and only one", but rather "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. Masculine pronouns (e.g., his) include feminine and neuter (e.g., her and its), and vice versa. Titles and subtitles (if any) are used merely for convenience and do not limit the present invention. The predicate words "configured to", "operable to", and "programmed to" do not imply any specific tangible or intangible modification to the subject matter, but are intended to be used interchangeably. For example, a processor configured to monitor and control an operation or component may also mean that the processor is programmed to monitor and control the operation, or that the processor is operable to monitor and control the operation. Likewise, a processor configured to execute code may be construed as a processor programmed to execute code or operable to execute code.

[0062] Phrases such as “aspects” do not imply that such aspects are essential to the subject technology or that such aspects apply to configurations of the subject technology. Disclosure relating to an aspect may apply to a configuration, or one or more configurations. An aspect may provide one or more examples. Phrases such as aspects may refer to one or more aspects, and vice versa. Phrases such as “embodiments” do not imply that such embodiments are essential to the subject technology or that such embodiments apply to configurations of the subject technology. Disclosure relating to an embodiment may apply to an embodiment, or one or more embodiments. An embodiment may provide one or more examples. Phrases such as “embodiments” may refer to one or more embodiments, and vice versa. Phrases such as “configurations” do not imply that such configurations are essential to the subject technology or that such configurations apply to configurations of the subject technology. Disclosure relating to a configuration may apply to a configuration, or one or more configurations. A configuration may provide one or more examples. Phrases such as “configurations” may refer to one or more configurations, and vice versa.

[0063] The word “exemplary” is used herein to mean “serving as an example or illustration.” Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0064] All structural and functional equivalents to the elements described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. Furthermore, to the extent that the terms "including," "having," and the like are used in the specification or claims, such terms are intended to be inclusive in a manner similar to the term "comprising," as interpreted when "including" is used as a transitional word in a claim.

[0065] References to "one embodiment," "an embodiment," "some embodiments," "various embodiments," etc., indicate that a particular element or feature is included in at least one embodiment of the present invention. Although these phrases may appear in various places, they do not necessarily refer to the same embodiment. In conjunction with the present invention, those skilled in the art will be able to design and incorporate any of a variety of mechanisms suitable for achieving the aforementioned functions.

[0066] It should be understood that the present invention teaches only one example of illustrative embodiments and that those skilled in the art will readily devise many variations of the invention upon reading this invention and that the scope of the invention is determined by the appended claims.

Claims

1. A joint training network device, comprising: Multi-head module, which includes: Network input; a feature network coupled to the network input and comprising a feature detector decoder that outputs an interest point and a descriptor generator decoder that outputs a descriptor, the feature detector decoder and the descriptor generator decoder being coupled in parallel; a depth network coupled to the network input and comprising a monocular depth prediction decoder and outputting a depth map; a stream network coupled to the network input and comprising an image segmentation decoder and outputting a segmented image; a segmentation network coupled to the network input and comprising a warp module that outputs a rotation and translation and an input warp signal to a segmentation decoder that outputs a residual stream; and A pose network is coupled to the network input and includes a fully connected pose estimator, the pose estimator is coupled to an adder, the adder receives input from the pose estimator and the residual stream from the segmentation decoder, the adder outputs an optical flow.

2. The joint training network device according to claim 1, wherein: The network input comes from a monocular camera.

3. The joint training network device according to claim 1, wherein: Also included are at least two joint training encoders, which receive a plurality of frames and output the network input.

4. The joint training network device according to claim 3, wherein: Also includes: A loop closure module comprising: a first loop closure encoder coupled to a first feature detector decoder that outputs a first interest point and a first descriptor generator decoder that outputs a first descriptor, the first feature detector decoder and the first descriptor generator decoder being coupled in parallel and outputting a first feature output; a second loop closure encoder coupled to a second feature detector decoder that outputs a second interest point and a second descriptor generator decoder that outputs a second descriptor, the second feature detector decoder and the second descriptor generator decoder being coupled in parallel and outputting a second feature output; and A fully connected similarity layer receives the first feature output and the second feature output and outputs a similarity.

5. The joint training network device according to claim 4, wherein: Also includes: A keyframe classifier module, which includes: a first key frame encoder having a first key frame encoder input receiving the plurality of frames and a first key frame encoder output; a second key frame encoder having a second key frame encoder input and a second key frame encoder output, wherein the second key frame encoder input is coupled to the first key frame encoder input and receives the plurality of frames, the second key frame encoder output is coupled in parallel to the first key frame encoder output, the second key frame encoder output is further coupled to a fully connected classifier layer, the fully connected classifier layer outputting a classified output; and A key frame selection switch is coupled to the first key frame encoder output and the classified output, wherein the key frame selection switch classifies a possible key frame in the plurality of frames as a key frame if a difference between a previous key frame and the possible key frame is greater than a predetermined threshold.

6. The joint training network device according to claim 5, wherein: Also includes: a first long short-term memory coupled in parallel to the first multi-head module and the keyframe classifier module; as well as A second long short-term memory is coupled to the output of the keyframe classifier module and is coupled in parallel to the second multi-head module and the loop closure module.

7. The joint training network device according to claim 6, wherein: The first stage trains the at least two joint training encoders; the second stage trains the at least two joint training encoders, the first long short-term memory, the multi-head module and the key frame classifier module; And the third stage trains the at least two joint training encoders, the second long short-term memory, the second stage multi-head module, the second stage key frame classifier module and the loop closure module.

8. The joint training network device according to claim 6, wherein: The first LSTM learns keyframe classification selection based on pose transitions and windowed contour-guided optimization, and the second LSTM learns pose dynamics and refines the global pose based on contour-guided optimization with loop closure.

9. The joint training network device according to claim 6, wherein: Also includes: A key frame detection module receives the first key frame encoder output and selects the key frame based on a difference between the first key frame encoder output and at least one of the first feature output and the second feature output.

10. The joint training network device according to claim 6, wherein: The second LSTM estimates a global pose by accumulating a set of predicted related poses from the first LSTM.

11. The joint training network device according to claim 6, wherein: The monocular depth prediction is performed by DepthNet.

12. The joint training network device according to claim 6, wherein: The image segmentation is performed by SegmentNet.

13. The joint training network device according to claim 6, wherein: The pose estimator is performed by PoseNet.

14. The joint training network device according to claim 6, wherein: The flow network is FlowNet.

Images