A simultaneous localization and mapping (SLAM) algorithm, a terminal and a storage medium

Abstract

The embodiment of the application discloses a kind of synchronous positioning and mapping SLAM algorithm, terminal and storage medium, terminal obtains training dataset;According to training dataset and initial network model, self-supervised training is handled, and target network model is obtained;Wherein, initial network model includes global attention module;Global attention module is used to learn the importance degree corresponding to different image blocks and the importance degree corresponding to different channels;Based on target network model, target image information is extracted and handled, and target transformation matrix is obtained;Based on target transformation matrix, point cloud data is constructed.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a simultaneous localization and mapping (SLAM) algorithm, terminal, and storage medium. Background Technology

[0002] Existing SLAM algorithms can be mainly divided into feature point methods and direct methods. Feature point methods first extract features and corresponding descriptors from the image. These features remain unchanged even with slight changes in camera viewpoint, allowing for data association between images through feature matching. When the obtained mapping is 2D to 2D, epipolar geometry can be used. For 3D to 2D mappings, Perspective-n-Point (PnP), Efficient Perspective-n-Point (EPnP), and Perspective Three-Point (P3P) algorithms can be used. For 3D to 3D mappings, Iterative Closest Point (ICP) algorithms can be used. After obtaining the transformation matrix, the camera pose and 3D coordinates of spatial points can be fine-tuned by calculating the reprojection error to obtain the final value. Direct methods, on the other hand, utilize the gray-level invariance assumption to perform data association from the optical flow of two frames, estimating the result by minimizing photometric error. Meanwhile, deep learning can also be applied to SLAM.

[0003] However, existing SLAM algorithms still have some problems. For the entire input image, not the information of each region block contributes to the estimation result. For example, dynamic objects in the environment will cause the model accuracy to decrease. In other words, existing SLAM algorithms generally have the defect of poor robustness. Summary of the Invention

[0004] This application provides a SLAM algorithm, terminal, and storage medium, which have strong robustness and accuracy.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a SLAM algorithm, the method comprising:

[0007] Obtain the training dataset;

[0008] The target network model is obtained by performing self-supervised training based on the training dataset and the initial network model; wherein, the initial network model includes a global attention module; the global attention module is used to learn the importance of different image patches and the importance of different channels;

[0009] Based on the target network model, feature extraction processing is performed on the target image information to obtain the target transformation matrix;

[0010] Point cloud data is constructed based on the target transformation matrix.

[0011] Secondly, embodiments of this application provide a terminal, which includes an acquisition unit, a training unit, and a construction unit.

[0012] The acquisition unit is used to acquire the training dataset;

[0013] The training unit is used to perform self-supervised training based on the training dataset and the initial network model to obtain the target network model; wherein, the initial network model includes a global attention module; the global attention module is used to learn the importance of different image patches and the importance of different channels;

[0014] The acquisition unit is used to perform feature extraction processing on the target image information based on the target network model to obtain the target transformation matrix;

[0015] The construction unit is used to construct point cloud data based on the target transformation matrix.

[0016] Thirdly, embodiments of this application provide a terminal, which includes a processor and a memory storing processor-executable instructions. When the instructions are executed by the processor, the SLAM algorithm described above is implemented.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program applied in a terminal, wherein the program, when executed by a processor, implements the SLAM algorithm as described above.

[0018] This application provides a SLAM algorithm, a terminal, and a storage medium. The terminal acquires a training dataset; performs self-supervised training based on the training dataset and an initial network model to obtain a target network model; wherein the initial network model includes a global attention module; the global attention module is used to learn the importance of different image patches and the importance of different channels; feature extraction processing is performed on the target image information based on the target network model to obtain a target transformation matrix; point cloud data is constructed based on the target transformation matrix. Therefore, this application, by adding a global attention module to the structure of the initial network model, enables the global attention module to learn the importance of different image patches and the importance of different channels, thereby allowing the target network model obtained after training to automatically "notice" the importance of different image regions during feature extraction processing, greatly improving model accuracy; furthermore, feature extraction processing is performed on the target image information based on the target network model, and point cloud data is constructed based on the obtained target transformation matrix, thus possessing strong robustness and accuracy. Attached Figure Description

[0019] Figure 1 This is a schematic diagram illustrating the implementation process of the SLAM algorithm proposed in the embodiments of this application;

[0020] Figure 2 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 1 ;

[0021] Figure 3 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 2 ;

[0022] Figure 4 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 3 ;

[0023] Figure 5 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 4 ;

[0024] Figure 6 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 5 ;

[0025] Figure 7 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 6 ;

[0026] Figure 8 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 7 ;

[0027] Figure 9This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 8 ;

[0028] Figure 10 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 9 ;

[0029] Figure 11 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 10 ;

[0030] Figure 12 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 10 one;

[0031] Figure 13 This is a schematic diagram of the terminal structure proposed in the embodiments of this application. Figure 1 ;

[0032] Figure 14 This is a schematic diagram of the terminal structure proposed in the embodiments of this application. Figure 2 . Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the accompanying drawings.

[0034] Real-time localization and environmental mapping of sensor-equipped machines during movement is a crucial problem in artificial intelligence (AI). Its main challenge is enabling machines to perceive, understand, and locate themselves in unfamiliar environments. In the AI ​​era, this technology can replace human intervention in certain specific scenarios, such as autonomous driving, mobile robots, and augmented reality (AR) / virtual reality (VR) applications. During localization, the machine's primary task is to perceive and characterize its surroundings. Many solutions exist for autonomous machine localization and environmental reconstruction based on known prior environmental information, such as the Global Positioning System (GPS), Inertial Navigation System (INS), and LiDAR systems. However, each has its drawbacks; for example, GPS can only be used outdoors, INS suffers from cumulative drift, and LiDAR systems are expensive. With the development of computer vision, researchers have focused on using only visual sensors (cameras) for localization and mapping without prior knowledge. Visual sensors are low-cost, high-precision, and can generate large amounts of image data for target detection and other environmental perception systems.

[0035] Traditional SLAM algorithms have developed relatively stable effects and implementation methods over the years, mainly divided into feature point methods and direct methods. Feature point methods first extract features and corresponding descriptors from the image. These features remain unchanged even with slight changes in camera viewpoint, allowing for data association between images through feature matching. For 2D-to-2D mappings, epipolar geometry can be used; for 3D-to-2D mappings, Perspective-n-Point (PnP), EfficientPerspective-n-Point (EPnP), and Perspective-Three-Point (P3P) algorithms can be used; and for 3D-to-3D mappings, Iterative Closest Point (ICP) algorithms can be used. After obtaining the transformation matrix, the camera pose and 3D coordinates of spatial points can be fine-tuned by calculating the reprojection error to obtain the final value. Direct methods, on the other hand, utilize the gray-level invariance assumption to perform data association from the optical flow of two frames, thus estimating the result by minimizing photometric error.

[0036] With the development of deep learning in recent years, it has also been applied to the field of SLAM. The basic idea of ​​supervised deep learning methods is to design a loss function that can guide the training of the network, and then train the network through a labeled dataset to regress the six degrees of freedom (6DOF) camera pose and pixel depth.

[0037] However, the traditional SLAM algorithms mentioned above still have some problems. For example, when the camera moves quickly or rotates at large angles, it may exceed the geometric calculation threshold set by the algorithm, resulting in "losing track". At the same time, these algorithms assume that the scene is static and do not have a good way to handle common dynamic objects in the environment, such as people and cars, which will cause a decrease in model accuracy. For example, traditional methods use manually designed shallow image features, which will reduce the accuracy of the algorithm or even make it impossible to run in challenging scenes such as drastic changes in lighting intensity, low brightness, and limited environmental texture. In addition, the geometric calculation is computationally intensive, which puts a heavy computational load on the system. The manually designed features are also relatively sparse, which cannot make full use of the information of the whole image.

[0038] Even existing supervised deep learning methods have their limitations. For example, when extracting features, the network typically receives the entire image as input. However, in real-world images, not every region contributes to the estimation results. For instance, regions with dynamic objects can decrease accuracy, and different regions may have long-distance coupling relationships. Typical networks cannot model this spatial information, leading to decreased model accuracy and generally poor robustness. Furthermore, supervised deep learning methods heavily rely on manually labeled datasets, which typically require significant human and material resources to obtain, increasing labor costs.

[0039] To address the problems existing in current Simultaneous Localization and Mapping (SLAM) algorithms, this application provides a SLAM algorithm, a terminal, and a storage medium. The terminal acquires a training dataset; performs self-supervised training based on the training dataset and an initial network model to obtain a target network model; wherein the initial network model includes a global attention module; the global attention module is used to learn the importance of different image patches and the importance of different channels; feature extraction processing is performed on the target image information based on the target network model to obtain a target transformation matrix; point cloud data is constructed based on the target transformation matrix; thereby enabling the SLAM algorithm to have strong robustness and accuracy.

[0040] It should be noted that this invention innovatively uses a global attention mechanism in the feature extraction stage of the SLAM network. This allows the model to automatically learn the dependencies between pixels, image blocks, and channels, thereby assigning importance weights to the spatial information of the extracted features. This allows for the reasonable utilization of spatial information in the image and features, thus improving the model's accuracy. The key point of the SLAM algorithm proposed in this application is its ability to extract better features. Because image features directly describe a single image, the geometric information contained between adjacent images provides data and a foundation for the overall computation process of the SLAM algorithm. Therefore, the quality of image features directly determines the effectiveness of localization and depth prediction. Thus, the global attention module used in this application to participate in feature extraction can greatly promote the model's results. This promotion is mainly reflected in three aspects: First, it can achieve higher accuracy. When extracting image features, the global attention module models the spatial information of channel correlation and long-distance dependencies, enabling the model to learn more image information, thereby improving accuracy. Second, it has strong robustness to challenging scenarios. Traditional SLAM algorithms rely on the static world assumption and use manually designed shallow image features, resulting in a significant drop in accuracy or even failure to run when faced with changing lighting or dynamic objects. In contrast, this application uses a deep learning-based neural network to extract high-dimensional image features, ensuring stable operation even in harsh environments. Thirdly, it reduces costs. The model uses affine transformation processing to achieve unsupervised training of the network, thus eliminating the need for extensive manually labeled datasets. Only continuous video sequences captured by visual sensors, such as cameras, are needed for network training.

[0041] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0042] Example 1

[0043] This application provides a transmission method. Figure 1 This is a schematic diagram illustrating the implementation process of the SLAM algorithm proposed in the embodiments of this application, as follows: Figure 1 As shown, the SLAM algorithm may include the following steps:

[0044] Step 101: Obtain the training dataset.

[0045] In the embodiments of this application, the terminal may first obtain the training dataset.

[0046] It should be noted that, in the embodiments of this application, the training dataset is used to train the initial network model, and the training dataset may be a dataset containing video sequences.

[0047] For example, in the embodiments of this application, the KITTI dataset is used as the training dataset. The KITTI dataset is currently the largest algorithm evaluation dataset for autonomous driving scenarios. This dataset contains real image data collected in urban areas, rural areas, and highways, and contains 22 video sequences, of which 11 video sequences have ground-truth labels. In this embodiment, the first 9 sequences 00-08 are used as the training dataset. In addition, this application can also determine the test dataset based on the KITTI dataset to test the target network model after training. For example, 09-10 in the KITTI dataset is selected as the test dataset to test the network performance.

[0048] Furthermore, in the embodiments of this application, in order to improve the speed and accuracy of training processing, the training dataset can be preprocessed, including size adjustment operations, etc.; the size adjustment operation can be to adjust the image size to a preset size, for example, to adjust the image size to 224×224; thereby better adapting to the training processing of the initial network model and meeting the real-time requirements of actual deployment.

[0049] Step 102: Perform self-supervised training based on the training dataset and the initial network model to obtain the target network model; wherein, the initial network model includes a global attention module; the global attention module is used to learn the importance of different image patches and the importance of different channels.

[0050] In the embodiments of this application, after obtaining the training dataset, the terminal can perform self-supervised training based on the training dataset and the initial network model to obtain the target network model; wherein, the initial network model includes a global attention module; the global attention module is used to learn the importance of different image patches and the importance of different channels.

[0051] It should be noted that, in the embodiments of this application, the initial network model includes a first sub-network and a second sub-network; the first sub-network can be a pose regression sub-network; and the second sub-network can be a depth prediction sub-network.

[0052] Furthermore, in the embodiments of this application, the feature extraction part of the first sub-network can be a ResNet network.

[0053] For example, in the embodiments of this application, Figure 2 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 1 ,like Figure 2The diagram shows five exemplary ResNet network architectures: ResNet18 (18-layer), ResNet34 (34-layer), ResNet50 (50-layer), ResNet101 (101-layer), and ResNet152 (152-layer). Taking ResNet18 as an example, the input consists of a 7×7 convolutional kernel and a 3×3 max-pooling layer, corresponding to the output size of the output layer. The size is 112×112; the 7×7 convolutional kernels have a dimension of 64 and a stride of 2; the stride of the 3×3 max pooling layer is 2; the middle part is mainly used for feature extraction and can include four convolutional blocks: conv2_x, conv3_x, conv4_x, and conv5_x; each convolutional block can be composed of stacked residual blocks, with the stacking times of the residual blocks being [2, 2, 2, 2], that is, conv2_x is composed of 2 stacked residual blocks; the output part consists of an average pooling layer, 1000-dimensional fully connected layers (FC), and a softmax function; different structures can correspond to different computational power (floating point operations per second, FLOPs), for example, ResNet18 has 1.8×10⁻⁶ FLOPs. 9 .

[0054] Furthermore, in embodiments of this application, the ResNet network includes an input portion, an output portion, and an intermediate portion; wherein the intermediate portion is composed of stacked residual blocks.

[0055] For example, in the embodiments of this application, Figure 3 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 2 ,like Figure 3 The diagram shows a basic-block. The input data is 64-dimensional and passes through two paths. One path goes through two 3×3 convolutions, and the other path is shorted. The two paths are added together and then output through the Rectified Linear Units (ReLU) function.

[0056] For example, in an embodiment of this application, the first sub-network can be constructed based on a ResNet18 network. Figure 4 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 3 ,like Figure 4The diagram shows the structure of the first sub-network using the ResNet18 network, which mainly includes an input stem, an intermediate part, and an output part. The intermediate part includes four convolutional blocks: Stage 1, Stage 2, Stage 3, and Stage 4. The input stem consists of a 7×7 convolutional kernel and a 3×3 max-pooling layer. The 7×7 convolutional kernel has a dimension of 64 and a stride of 2. The stride of the 3×3 max-pooling layer is 2. Stage 4 includes downsampling and residual blocks. The output stem can implement globally adaptive smooth pooling, for example, it can transform a 1×512×7×7 dimensional input data into a 1×512×1×1 dimensional input to obtain the final features used to regress the 6D0F pose.

[0057] Furthermore, in the embodiments of this application, the depth prediction sub-network is a U-Net network; the U-Net network includes an encoder and a decoder; the encoder can be a ResNet network; the decoder can be a DispNet network.

[0058] For example, in the embodiments of this application, the depth prediction subnetwork is a U-Net network; the encoder of the depth prediction subnetwork can be a ResNet50 network, and the decoder can be a DispNet network; wherein, the basic structure of ResNet50 is the same as that of ResNet18, only the number of convolutional blocks stacked in the middle is different. Figure 5 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 4 ,like Figure 5 The diagram shows the U-Net network structure. During forward propagation, the network performs backward deconvolution (unconv) while simultaneously predicting depth directly on small feature maps. The results are then bilinearly interpolated and concatenated onto the unconv feature map. This process is repeated four times, resulting in a predicted depth resolution that is 1 / 4 of the input. Furthermore, bilinear interpolation is used to obtain the previous frame image I from the adjacent input frames. t Depth map D at the same resolution t .

[0059] Furthermore, in the embodiments of this application, during training, adjacent frame images from the training dataset are input into the initial network model; when inputting adjacent frame images, two adjacent frame images can be stitched together according to channels; for example, two adjacent frame images can be stitched together to form image data of size 256×256×6.

[0060] It is understood that, in the embodiments of this application, the target network model refers to the network model obtained after the initial network model has been trained.

[0061] It should be noted that in the embodiments of this application, a global attention module is added to the initial network model. Compared with ordinary networks in the prior art, ordinary networks treat every pixel and every channel on the feature map extracted by the encoder and the original image equally. This approach has significant drawbacks. For example, in real images, not every region block's information contributes to the SLAM result. Regions with dynamic objects can cause a decrease in accuracy, and different regions of image blocks have long-distance coupling relationships. By adding a global attention module, the model can automatically learn the long-distance coupling relationships between image blocks and automatically assign importance weights to the information of different image blocks. This allows the model to automatically "notice" the importance of different regions in the image, improving the model's accuracy. The global attention module can also implement channel weighting, thereby establishing the correlation of spatial information. Adding a global attention module to the feature extraction part of the pose regression subnetwork and the depth prediction subnetwork can better extract features, thereby improving the model's accuracy and robustness.

[0062] Furthermore, in the embodiments of this application, the global attention module is a global context attention (GC) module; the global attention module includes a first attention module and a second attention module.

[0063] Furthermore, in the embodiments of this application, the first sub-network includes a first attention module; the second sub-network includes a second attention module; that is, this application adds the first attention module to the first sub-network and adds the second attention module to the second sub-network, so that the first sub-network and the second sub-network can further perform spatial information fusion and weighting on the features when performing feature extraction processing.

[0064] For example, in the embodiments of this application, the first sub-network is a pose regression sub-network, the second sub-network is a deep prediction sub-network, and the global attention module is a GC module; the first attention module is added after the feature extraction part of the pose regression sub-network, that is, after the residual block in the feature extraction part is finished; the second attention module is added to the encoder of the deep prediction sub-network, for example, after the residual block in the encoder is finished.

[0065] It should be noted that commonly used attention mechanisms include channel attention (Squeeze-and-Excitation, SE) and spatial attention (Simplified Non-local Block, SNL). Figure 6 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 5 ,like Figure 6 The diagram shows the structure of the channel attention mechanism; where the symbol "⊙" represents broadcast element-wise multiplication; the channel attention mechanism allows the model to learn the importance of different channels and thus weight them accordingly. Figure 7 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 6 ,like Figure 7 The diagram shows the structure of the spatial attention mechanism; where the symbol "⊕" represents broadcast element addition; in the spatial attention mechanism, the transformation part has a large number of parameters, employing a 1×1 convolutional layer W with C×C parameters. v Spatial attention can establish the correlation of global spatial information by modeling long-distance dependencies and weighted fusion of information from distant locations.

[0066] Furthermore, embodiments of this application employ a Global Context Network (GC) module. Figure 8 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 7 ,like Figure 8 The diagram shows the structure of the global context attention module; where, symbols This represents matrix multiplication. In the global context attention module, a 1×1 convolutional kernel softmax function is first used to obtain attention weights, which are then multiplied with the original input features to obtain global context features. Unlike the spatial attention mechanism, the global context attention module uses a bottleneck transformation module instead of the convolutional layer W. v The bottleneck transformation module can establish channel dependencies, reducing the number of parameters from C×C to 2×C×C / r, where r is the bottleneck ratio, with a default setting of r=16. Compared to the spatial attention mechanism, this reduces the number of parameters in the transformation module to 1 / 8 of that in the spatial attention mechanism. Since the bottleneck transformation module of the global context attention module increases the optimization difficulty, a normalization layer is added before the bottleneck transformation to facilitate optimization. The global context attention module has both spatial global context modeling capabilities and can implement channel weighting to establish spatial information correlations, while having a small number of parameters and low computational complexity. In other words, the global context attention module includes global attention computation for context modeling, channel dependencies modeled using bottleneck transformation, and feature fusion with the original input in the form of broadcast elements. Furthermore, because the global context attention module is lightweight, it can be applied to multiple layers of the network to better capture long-range dependencies. The global context attention module can be expressed as the following formula:

[0067]

[0068] in, It is the global attention weight α j ,Right now δ(·)=W v2 ReLU{LN{W v1 (·)}} indicates a bottleneck transition.

[0069] It should be noted that, in the embodiments of this application, the GC module mainly functions in three ways: first, it is used for context modeling to obtain global context features; second, it establishes channel dependencies based on the bottleneck transformation module; and third, it performs feature fusion with the original input features in the form of broadcast elements. Furthermore, because the GC module is lightweight, it can be applied to multiple layers in the network, enabling it to better capture remote dependencies.

[0070] In some embodiments of this application, the method for obtaining a target network model by performing self-supervised training on the terminal based on the training dataset and the initial network model may include: obtaining a depth map and a transformation matrix based on adjacent frame images in the training dataset and the initial network model; wherein, the depth map represents the depth information of the previous frame image in the adjacent frame images; the transformation matrix represents the relative transformation relationship between adjacent frame images; then, based on the depth map and the transformation matrix, a predicted image corresponding to the next frame image in the adjacent frame images is obtained, and a loss function value is calculated based on the predicted image; and the initial network model is subjected to self-supervised training based on the loss function value to obtain the target network model.

[0071] Step 103: Perform feature extraction processing on the target image information based on the target network model to obtain the target transformation matrix.

[0072] In the embodiments of this application, after the terminal performs self-supervised training based on the training dataset and the initial network model to obtain the target network model, it can perform feature extraction processing on the target image information based on the target network model to obtain the target transformation matrix.

[0073] It should be noted that, in the embodiments of this application, target image information refers to the image information input when constructing point cloud data using a target network model; target image information can be a large amount of image data acquired by a visual sensor.

[0074] It is understood that in the embodiments of this application, when performing feature extraction processing on target image information based on the target network model, the feature extraction processing is still based on two adjacent frames of the target image information to obtain the corresponding target transformation matrix.

[0075] It should be noted that, in the embodiments of this application, the target transformation matrix is ​​the estimated target of SLAM; the target transformation matrix can characterize the relative transformation relationship between adjacent frame image information in the target image information.

[0076] Step 104: Construct point cloud data based on the target transformation matrix.

[0077] In the embodiments of this application, after the terminal performs feature extraction processing on the target image information based on the target network model to obtain the target transformation matrix, it can construct point cloud data based on the target transformation matrix.

[0078] It should be noted that, in the embodiments of this application, point cloud data is a collection of sampling points with spatial coordinates. It is large in number and dense, and can be widely used in surveying, power, construction, industry and other fields.

[0079] Furthermore, in embodiments of this application, the method for a terminal to perform feature extraction processing on target image information based on a target network model may include the following steps:

[0080] Step 201: Perform global attention calculation on the target image information based on the target global attention module in the target network model to complete the feature extraction process.

[0081] In some embodiments of this application, the terminal performs feature extraction processing on the target image information based on the target network model. In some embodiments of this application, the terminal can perform global attention calculation on the target image information based on the target global attention module in the target network model to complete the feature extraction processing.

[0082] It should be noted that, in the embodiments of this application, the target global attention module refers to the global attention module contained in the target network model obtained after training.

[0083] It is understood that, in the embodiments of this application, since the target network model includes a target global attention module, the process of using the target network model to perform feature extraction processing on the target image information also includes a calculation process involving the target global attention module, which is the global attention calculation.

[0084] Furthermore, in the embodiments of this application, global attention calculation is performed using the target global attention module to obtain the enhanced features corresponding to the original input features in the input global attention module.

[0085] Furthermore, in the embodiments of this application, the method by which the terminal performs global attention calculation on target image information based on the target global attention module in the target network model, i.e., the method proposed in step 201, may include the following steps:

[0086] Step 201a: Obtain attention weights through the first module in the target global attention module, and multiply the attention weights with the original input features to obtain global context features.

[0087] In the embodiments of this application, the terminal performs global attention calculation on the target image information based on the target global attention module in the target network model. In some embodiments of this application, the terminal can first obtain the attention weight through the first module in the target global attention module, and then multiply the attention weight with the original input features to obtain the global context features.

[0088] It should be noted that, in the embodiments of this application, the structure of the first module is as described above. Figure 8 As shown, it can include a 1×1 convolutional layer and a softmax function.

[0089] Furthermore, in the embodiments of this application, attention weights can characterize the importance of different regions in the target image information.

[0090] Furthermore, in the embodiments of this application, as described above... Figure 8 As shown, after obtaining the attention weights through the first module containing a 1×1 convolutional layer and a softmax function, the attention weights are multiplied with the original input features corresponding to the target image information to obtain global context features.

[0091] Step 201b: Obtain the channel dependency relationship corresponding to the global context feature according to the second module in the global attention module; wherein, the second module includes a bottleneck transformation module, a normalization layer and an activation layer.

[0092] In the embodiments of this application, after the terminal obtains attention weights through the first module in the target global attention module and multiplies the attention weights with the original input features to obtain global context features, it can obtain the channel dependency relationship corresponding to the global context features according to the second module in the global attention module; wherein, the second module includes a bottleneck transformation module, a normalization layer and an activation layer.

[0093] It should be noted that, in the embodiments of this application, the second module includes a bottleneck conversion module, a normalization layer, and an activation layer; for example, as described above. Figure 8 As shown, the second module includes a bottleneck transformation module, namely two 1×1 convolutional layers conv(1×1), a normalization layer LayerNorm, and an activation layer ReLU.

[0094] Therefore, in the embodiments of this application, by inputting global context features into the second module, the second module can output channel dependencies.

[0095] Step 201c: Perform feature fusion processing based on channel dependency and target image information to complete global attention calculation.

[0096] In the embodiments of this application, after the terminal obtains the channel dependency relationship corresponding to the global context features according to the second module, it can perform feature fusion processing based on the channel dependency relationship and the target image information to complete the global attention calculation.

[0097] For example, in the embodiments of this application, as described above Figure 8 As shown, after obtaining the channel dependencies, feature fusion processing is performed by broadcast element addition based on the channel dependencies and the original input, i.e., the target image information, thereby completing the global attention calculation.

[0098] This application provides a SLAM algorithm. The terminal acquires a training dataset; self-supervised training is performed on the training dataset and an initial network model to obtain a target network model; the initial network model includes a global attention module; the global attention module learns the importance of different image patches and the importance of different channels; feature extraction is performed on the target image information based on the target network model to obtain a target transformation matrix; point cloud data is constructed based on the target transformation matrix. Therefore, by adding a global attention module to the structure of the initial network model, this application enables the global attention module to learn the importance of different image patches and the importance of different channels. This allows the target network model, after training, to automatically "notice" the importance of different image regions during feature extraction, greatly improving model accuracy; furthermore, feature extraction is performed on the target image information based on the target network model, and point cloud data is constructed based on the obtained target transformation matrix, thus exhibiting strong robustness and accuracy.

[0099] Example 2

[0100] Based on the above embodiments, in another embodiment of this application, the method for obtaining the target network model by performing self-supervised training on the terminal according to the training dataset and the initial network model, i.e., the method proposed in step 102, may include the following steps:

[0101] Step 102a: Obtain a depth map and a transformation matrix based on adjacent frame images in the training dataset and the initial network model; wherein, the depth map represents the depth information of the previous frame image in adjacent frame images; the transformation matrix represents the relative transformation relationship between adjacent frame images.

[0102] In the embodiments of this application, the terminal performs self-supervised training based on the training dataset and the initial network model to obtain the target network model. In some embodiments of this application, the terminal can first obtain a depth map and a transformation matrix based on adjacent frame images in the training dataset and the initial network model; wherein, the depth map represents the depth information of the previous frame image in adjacent frame images; and the transformation matrix represents the relative transformation relationship between adjacent frame images.

[0103] It is understood that, in the embodiments of this application, adjacent frame images include the previous frame image and the image of the next frame corresponding to the previous frame image; by inputting adjacent frame images from the training dataset into the initial network model, the corresponding depth map and transformation matrix can be obtained.

[0104] It should be noted that, in the embodiments of this application, the depth map represents the depth information of the previous frame image in adjacent frame images.

[0105] It should also be noted that, in the embodiments of this application, the transformation matrix represents the relative transformation relationship between adjacent frame images in the training dataset.

[0106] Step 102b: Obtain the predicted image corresponding to the next frame image in the adjacent frame images based on the depth map and transformation matrix, calculate the loss function value based on the predicted image, and perform self-supervised training on the initial network model based on the loss function value to obtain the target network model.

[0107] In the embodiments of this application, after the terminal obtains the depth map and transformation matrix based on the adjacent frame images and the initial network model in the training dataset, it can obtain the predicted image corresponding to the next frame image in the adjacent frame images according to the depth map and transformation matrix, calculate the loss function value based on the predicted image, and perform self-supervised training processing on the initial network model according to the loss function value to obtain the target network model.

[0108] In some embodiments of this application, a method for a terminal to obtain a predicted image corresponding to the next frame image in adjacent frame images based on a depth map and a transformation matrix may include: performing affine transformation processing based on camera intrinsic information, a depth map, and a transformation matrix to obtain a predicted image; wherein, affine transformation processing also refers to warping calculation.

[0109] In other words, in the embodiments of this application, the terminal mainly obtains the predicted image corresponding to the next frame image through affine transformation processing, and then uses the predicted image to calculate the loss function value, thereby performing self-supervised training processing on the initial network model based on the loss function value; it can achieve joint optimization without manual annotation, reduce the model's dependence on supervised data, and effectively reduce training costs.

[0110] Furthermore, in the embodiments of this application, the predicted image represents the prediction of the next frame image in the adjacent frame images, and can also be understood as the predicted image being a "fake" next frame image in the adjacent frame images.

[0111] Furthermore, in the embodiments of this application, the loss function value can be used to perform self-supervised training on the initial network model; the loss function value includes a first loss function value and a second loss function value.

[0112] In some embodiments of this application, the method for a terminal to calculate a loss function value based on a predicted image may include: calculating a first loss function value based on the predicted image and a subsequent frame image in adjacent frame images; wherein the first loss function value characterizes the degree of difference between the subsequent frame image in adjacent frame images and the predicted image; calculating a second loss function value based on a depth map; wherein the second loss function value characterizes the degree of change in the depth map; and determining a loss function value based on the first loss function value and the second loss function value.

[0113] It is understood that in the embodiments of this application, the terminal performs self-supervised training on the initial network model based on the loss function value, that is, it guides the subsequent training direction based on the loss function value; for example, the loss function value can be backpropagated along the direction of minimum gradient based on the derivative of the loss function, thereby correcting the weight values ​​in the forward calculation formula until the loss function value meets the preset threshold, at which point the training process can be stopped.

[0114] Furthermore, in the embodiments of this application, the method by which the terminal obtains the predicted image corresponding to the next frame image in adjacent frame images based on the depth map and the transformation matrix, i.e., the method proposed in step 102b, may include the following steps:

[0115] Step 102b1: Perform affine transformation processing based on camera intrinsic information, the depth map, and the transformation matrix to obtain the predicted image.

[0116] In some embodiments of this application, the terminal obtains the predicted image corresponding to the next frame image in adjacent frame images based on the depth map and the transformation matrix. In some embodiments of this application, the terminal can perform affine transformation processing based on camera intrinsic information, the depth map and the transformation matrix to obtain the predicted image.

[0117] For example, in the embodiments of this application, the principle of affine transformation processing, i.e., warping calculation, can be expressed as: p t+1 ~KT t→s D t (p t )K -1 p t Where t and s represent the next and previous frames in the adjacent input images, respectively, p represents the pixel position, and K, T, and D represent the camera intrinsic parameters, transformation matrix, and depth map, respectively. This embodiment of the application obtains the predicted image corresponding to the next frame image through affine transformation processing.

[0118] Furthermore, in embodiments of this application, the method for the terminal to calculate the loss function value based on the predicted image may include the following steps:

[0119] Step 301: Calculate the first loss function value based on the predicted image and the next frame image in the adjacent frame images; wherein, the first loss function value characterizes the degree of difference between the next frame image in the adjacent frame images and the predicted image.

[0120] In some embodiments of this application, the terminal calculates a loss function value based on the predicted image. In some embodiments of this application, the terminal can calculate a first loss function value based on the predicted image and the next frame image in the adjacent frame images; wherein, the first loss function value characterizes the degree of difference between the next frame image in the adjacent frame images and the predicted image.

[0121] It should be noted that, in the embodiments of this application, the first loss function value is calculated based on the predicted image and the next frame image in the adjacent frame images. In other words, the first loss function is calculated based on the predicted next frame image in the adjacent frame images and the actual next frame image in the adjacent frame images. Therefore, the first loss function value can characterize the degree of difference between the next frame image in the adjacent frame images and the predicted image.

[0122] Furthermore, in the embodiments of this application, the calculation method of the first loss function may include two methods, one of which can be expressed as the following formula:

[0123]

[0124] Where N is the number of pixels, and I(t+1) represents the actual next frame image. This represents the predicted image corresponding to the next frame.

[0125] Furthermore, in the embodiments of this application, in order to better measure the difference between the real next frame image and the predicted next frame image, the embodiments of this application may also adopt another calculation method: using the Structural Similarity Index (SSIM) to calculate the first loss function value.

[0126] Step 302: Calculate the second loss function value based on the depth map; whereby the second loss function value characterizes the degree of change in the depth map.

[0127] In some embodiments of this application, a loss function value is calculated based on the predicted image. In some embodiments of this application, the terminal can calculate a second loss function value based on the depth map; wherein, the second loss function value characterizes the degree of change in the depth map.

[0128] It should be noted that, in the embodiments of this application, in order to make the obtained depth map have the property of local smoothness, a smoothness error can be used to constrain and optimize the depth map, that is, a second loss function is calculated based on the depth map.

[0129] For example, in an embodiment of this application, the method for determining the value of the second loss function can be expressed as the following formula:

[0130]

[0131] Where I(x,y) represents the depth map and D(x,y) represents the pixel with coordinates (x,y).

[0132] Furthermore, in the embodiments of this application, the second loss function value characterizes the degree of change in the depth map.

[0133] Step 303: Determine the loss function value based on the first loss function value and the second loss function value.

[0134] In the embodiments of this application, after the terminal calculates a first loss function value based on the predicted image and the next frame image in the adjacent frame images, and calculates a second loss function value based on the depth map, it can determine the loss function value based on the first loss function value and the second loss function value.

[0135] It should be noted that, in the embodiments of this application, the first loss function value and the second loss function value can be assigned corresponding weight parameters respectively, and the weight parameters can be multiplied by the first loss function value and the second loss function value respectively. The product of the first loss function value and the weight parameters is then added to the product of the second loss function value and the weight parameters to obtain the loss function value.

[0136] For example, in an embodiment of this application, the loss function value determined based on the first loss function value and the second loss function value is:

[0137] LOSS = αL photo +βL smooth (4)

[0138] Here, α and β are the weight parameters corresponding to the first loss function value and the second loss function value, respectively.

[0139] Furthermore, in the embodiments of this application, the method by which the terminal calculates the first loss function value based on the predicted image and the next frame image in the adjacent frame images, i.e., the method proposed in step 301, may include the following steps:

[0140] Step 301a: Calculate the first loss function value based on the predicted image, the next frame image in the adjacent frame images, and the structural similarity index SSIM.

[0141] In some embodiments of this application, the terminal calculates a first loss function value based on the predicted image and the next frame image in the adjacent frame images. In some embodiments of this application, the terminal can calculate the first loss function value based on the predicted image, the next frame image in the adjacent frame images, and SSIM.

[0142] For example, the first loss function value calculated using SSIM can be expressed as the following formula:

[0143]

[0144] in, This refers to the portion calculated using SSIM; I(t+1) represents the predicted image; I(t+1) represents the next frame image in the adjacent frame images.

[0145] Furthermore, in the embodiments of this application, the method by which the terminal obtains the depth map and transformation matrix based on adjacent frame images and the initial network model in the training dataset, i.e., the method proposed in step 102a, may include the following steps:

[0146] Step 102a1: Perform feature extraction processing on adjacent frame images based on the first sub-network to obtain the transformation matrix.

[0147] In some embodiments of this application, the terminal obtains a depth map and a transformation matrix based on adjacent frame images and an initial network model in the training dataset. In some embodiments of this application, the terminal can perform feature extraction processing on the previous frame image based on the first sub-network to obtain a depth map.

[0148] For example, in an embodiment of this application, the first sub-network is a pose regression sub-network built on ResNet18. The pose regression sub-network is used to perform feature extraction processing on two adjacent frame images to obtain a transformation matrix.

[0149] Step 102a2: Perform feature extraction processing on the previous frame image based on the second sub-network to obtain a depth map.

[0150] In some embodiments of this application, the terminal obtains a depth map and a transformation matrix based on adjacent frame images in the training dataset and an initial network model. In some embodiments of this application, the terminal can perform feature extraction processing on the previous frame image based on a second sub-network to obtain a depth map.

[0151] For example, in an embodiment of this application, the second sub-network is a depth prediction sub-network of a U-Net network structure. The depth prediction sub-network is used to perform feature extraction processing on the previous frame image in adjacent frame images to obtain a depth map.

[0152] In summary, in the embodiments of this application, exemplarily, Figure 9 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 8 ,like Figure 9The diagram illustrates the training principle of the initial network model. The input consists of image information from two adjacent frames: I(t) and I(t+1). The image information from the previous frame, I(t), is input into the second sub-network, i.e., the depth prediction sub-network. I(t) and I(t+1) are input into the first sub-network, i.e., the pose regression sub-network. The feature extraction part of the first sub-network can use a ResNet18 network. The second sub-network includes an encoder and a decoder. The encoder in the second sub-network contains a GC module. The second sub-network also includes a global attention module, placed after the feature extraction module of PoseNet. The second sub-network also includes a fully connected layer (FC). The depth map D(T) is obtained from the second sub-network, and the transformation matrix T is obtained from the first sub-network. Finally, the second loss function L is obtained based on the depth map D(T). smooth Simultaneously, affine transformation processing (warping) is performed based on the transformation matrix T and the depth map D(T) to obtain the predicted image. And based on The first loss function L is obtained by performing the corresponding calculations. photo Therefore, based on L smooth and L photo Determine the loss function and perform self-supervised training based on the loss function.

[0153] Furthermore, in the embodiments of this application, the method by which the terminal performs feature extraction processing on the target image information based on the target network model to obtain the target transformation matrix, i.e., the method proposed in step 103, may include the following steps:

[0154] Step 103a: Based on the first sub-network of the target network model, perform feature extraction processing on the target image information to obtain the initial transformation matrix.

[0155] In some embodiments of this application, the terminal performs feature extraction processing on the target image information based on the target network model to obtain the target transformation matrix. In some embodiments of this application, the terminal can perform feature extraction processing on the target image information based on the first target sub-network in the target network model to obtain the initial transformation matrix.

[0156] It should be noted that, in the embodiments of this application, the first target sub-network is based on feature extraction processing of two adjacent frames of images in the target image information.

[0157] It is understood that in the embodiments of this application, the target first sub-network is the trained first sub-network; for example, the target first sub-network can be the trained pose regression sub-network.

[0158] It should be noted that, in the embodiments of this application, the initial transformation matrix refers to the matrix information obtained after performing feature extraction processing on the target image information using the first target sub-network.

[0159] Step 103b: Based on the target second sub-network in the target network model, perform feature extraction processing on the previous frame image in the target image information to obtain the target depth map corresponding to the previous frame image in the target image information.

[0160] In some embodiments of this application, the terminal performs feature extraction processing on the target image information based on the target network model to obtain the target transformation matrix. In some embodiments of this application, the terminal can perform feature extraction processing on the previous frame image in the target image information based on the target second sub-network in the target network model to obtain the target depth map corresponding to the previous frame image in the target image information.

[0161] It should be noted that, in the embodiments of this application, the target second sub-network is the trained second sub-network; for example, the target second sub-network is the trained deep prediction sub-network.

[0162] It should be noted that, in the embodiments of this application, the target depth map is a depth map obtained by performing feature extraction processing on the previous frame image in the target image information using the second target sub-network; the target depth map can represent the depth information of the previous frame image in the target image information.

[0163] Step 103c: Optimize the initial transformation matrix using the target depth map to obtain the target transformation matrix.

[0164] In the embodiments of this application, after the terminal performs feature extraction processing on the target image information based on the first target sub-network in the target network model to obtain an initial transformation matrix, and performs feature extraction processing on the previous frame image in the target image information based on the second target sub-network in the target network model to obtain a target depth map corresponding to the previous frame image in the target image information, the terminal can use the target depth map to optimize the initial transformation matrix to obtain a target transformation matrix.

[0165] It should be noted that, in the embodiments of this application, the target depth map can be used to optimize the initial transformation matrix, which can make the obtained target transformation matrix more accurate, thereby more accurately describing the relative transformation relationship between adjacent frame images in the target image information.

[0166] Furthermore, in the embodiments of this application, in order to evaluate the performance of the target network model, this application can use a test dataset to test the overall motion trajectory and depth value of the target network model; Figure 10 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 9,like Figure 10 The image shows a specific test example, which is the motion trajectory result of the test using the 09 sequence in the KITTI dataset; Figure 11 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 10 ,like Figure 11 The image shows the motion trajectory results tested using 10 sequences from the KITTI dataset; where Ground Truth is the label, representing the real data; Ours represents the data obtained using the target network model in this application embodiment. Figure 12 This is a schematic diagram of the implementation of the SLAM algorithm proposed in the embodiments of this application. Figure 10 First, such as Figure 12 The image shown is a depth map for a single frame. As can be seen, this application addresses challenging scenarios with dynamic objects by adding a global attention module. This module captures global spatial context information, applies attention weights to channels, and has low computational complexity, resulting in a more accurate and robust target network model.

[0167] This application provides a SLAM algorithm, a terminal, and a storage medium. The terminal acquires a training dataset; performs self-supervised training based on the training dataset and an initial network model to obtain a target network model; wherein the initial network model includes a global attention module; the global attention module is used to learn the importance of different image patches and the importance of different channels; feature extraction processing is performed on the target image information based on the target network model to obtain a target transformation matrix; point cloud data is constructed based on the target transformation matrix. Therefore, this application, by adding a global attention module to the structure of the initial network model, enables the global attention module to learn the importance of different image patches and the importance of different channels, thereby allowing the target network model obtained after training to automatically "notice" the importance of different image regions during feature extraction processing, greatly improving model accuracy; furthermore, feature extraction processing is performed on the target image information based on the target network model, and point cloud data is constructed based on the obtained target transformation matrix, thus possessing strong robustness and accuracy.

[0168] Example 3

[0169] Based on the above embodiments, in another embodiment of this application... Figure 13 This is a schematic diagram of the terminal structure proposed in the embodiments of this application. Figure 1 ,like Figure 13 As shown, the terminal 10 proposed in this embodiment may include an acquisition unit 11, a training unit 12, an acquisition unit 13, and a construction unit 14.

[0170] The acquisition unit 11 is used to acquire the training dataset.

[0171] The training unit 12 is used to perform self-supervised training based on the training dataset and the initial network model to obtain the target network model; wherein, the initial network model includes a global attention module; the global attention module is used to learn the importance of different image patches and the importance of different channels.

[0172] The acquisition unit 13 is used to perform feature extraction processing on the target image information based on the target network model to obtain the target transformation matrix.

[0173] The construction unit 14 is used to construct point cloud data based on the target transformation matrix.

[0174] Furthermore, the global attention module is a global context attention (GC) module; the global attention module includes a first attention module and a second attention module.

[0175] Furthermore, the acquisition unit 13 is also used to perform global attention calculation on the target image information based on the target global attention module in the target network model, so as to complete the feature extraction process.

[0176] Furthermore, the acquisition unit 13 is also used to acquire attention weights through the first module in the target global attention module, multiply the attention weights by the original input features to obtain global context features; and acquire the channel dependency relationship corresponding to the global context features according to the second module in the target global attention module; wherein the second module includes a bottleneck transformation module, a normalization layer and an activation layer; and perform feature fusion processing with the target image information according to the channel dependency relationship to complete the global attention calculation.

[0177] Furthermore, the initial network model includes a first sub-network and a second sub-network; the first sub-network includes the first attention module; and the second sub-network includes the second attention module.

[0178] Furthermore, the first sub-network is a pose regression sub-network; the feature extraction module of the pose regression sub-network is a ResNet network.

[0179] Furthermore, the second sub-network is a deep prediction sub-network; the deep prediction sub-network is a U-Net network; the U-Net network includes an encoder and a decoder; the encoder is a ResNet network; and the decoder is a DispNet network.

[0180] Furthermore, the training unit 12 is also used to obtain a depth map and a transformation matrix based on adjacent frame images in the training dataset and the initial network model; wherein, the depth map represents the depth information of the previous frame image in the adjacent frame images; and the transformation matrix represents the relative transformation relationship between the adjacent frame images.

[0181] The predicted image corresponding to the next frame in adjacent frame images is obtained based on the depth map and the transformation matrix. The loss function value is calculated based on the predicted image. The initial network model is then subjected to self-supervised training based on the loss function value to obtain the target network model.

[0182] Furthermore, the training unit 12 is also used to perform affine transformation processing based on camera intrinsic information, the depth map, and the transformation matrix to obtain the predicted image.

[0183] Furthermore, the training unit 12 is also configured to calculate a first loss function value based on the predicted image and the next frame image in the adjacent frame images; wherein the first loss function value characterizes the degree of difference between the next frame image in the adjacent frame images and the predicted image; and calculate a second loss function value based on the depth map; wherein the second loss function value characterizes the degree of change in the depth map; and determine the loss function value according to the first loss function value and the second loss function value.

[0184] Furthermore, the training unit 12 is also used to calculate the first loss function value based on the predicted image, the next frame image in the adjacent frame images, and the structural similarity index SSIM.

[0185] Furthermore, the training unit 12 is also used to perform feature extraction processing on the adjacent frame images based on the first sub-network to obtain the transformation matrix;

[0186] The depth map is obtained by performing feature extraction processing on the previous frame image based on the second sub-network.

[0187] Furthermore, the ResNet network includes an input section, an output section, and an intermediate section; wherein the intermediate section is composed of stacked residual blocks.

[0188] Furthermore, the acquisition unit 13 is also used to perform feature extraction processing on the target image information based on the target first sub-network in the target network model to obtain an initial transformation matrix; and to perform feature extraction processing on the previous frame image in the target image information based on the target second sub-network in the target network model to obtain a target depth map corresponding to the previous frame image in the target image information; and to optimize the initial transformation matrix using the target depth map to obtain the target transformation matrix.

[0189] Figure 14 This is a schematic diagram of the terminal structure proposed in the embodiments of this application. Figure 2 ,like Figure 14 As shown, the terminal proposed in this application embodiment may further include a processor 15, a memory 16 storing instructions executable by the processor 15, and further, the terminal 20 may further include a communication interface 17 and a bus 18 for connecting the processor 15, the memory 16 and the communication interface 17.

[0190] In the embodiments of this application, the processor 15 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and this application embodiment does not specifically limit it. The processor 15 may also include a memory 16, which can be connected to the processor 15. The memory 16 is used to store executable program code, which includes computer operation instructions. The memory 16 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.

[0191] In embodiments of this application, bus 18 is used to connect communication interface 17, processor 15, and memory 16, as well as the mutual communication between these devices.

[0192] In embodiments of this application, memory 16 is used to store instructions and data.

[0193] Furthermore, in an embodiment of this application, the processor 15 is used to acquire a training dataset;

[0194] The target network model is obtained by performing self-supervised training based on the training dataset and the initial network model; wherein, the initial network model includes a global attention module; the global attention module is used to learn the importance of different image patches and the importance of different channels;

[0195] Based on the target network model, feature extraction processing is performed on the target image information to obtain the target transformation matrix;

[0196] Point cloud data is constructed based on the target transformation matrix.

[0197] In practical applications, the aforementioned memory 16 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 15.

[0198] Furthermore, in this embodiment, the functional modules can be integrated into one analysis unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0199] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0200] This application provides a terminal and storage medium. The terminal acquires a training dataset; performs self-supervised training based on the training dataset and an initial network model to obtain a target network model; wherein the initial network model includes a global attention module; the global attention module is used to learn the importance of different image patches and the importance of different channels; feature extraction processing is performed on the target image information based on the target network model to obtain a target transformation matrix; point cloud data is constructed based on the target transformation matrix. Therefore, by adding a global attention module to the structure of the initial network model, this application enables the global attention module to learn the importance of different image patches and the importance of different channels, thereby allowing the target network model obtained after training to automatically "notice" the importance of different image regions during feature extraction processing, greatly improving model accuracy; furthermore, feature extraction processing is performed on the target image information based on the target network model, and point cloud data is constructed based on the obtained target transformation matrix, thus exhibiting strong robustness and accuracy.

[0201] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0202] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0203] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0204] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0205] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A Simultaneous Localization and Mapping (SLAM) method, characterized in that, The method includes: Obtain the training dataset; The target network model is obtained by performing self-supervised training based on the training dataset and the initial network model. The initial network model includes a global attention module, which learns the importance of different image patches and different channels. The global attention module includes a first attention module and a second attention module. The initial network model includes a first sub-network and a second sub-network. The first sub-network includes the first attention module, and the second sub-network includes the second attention module. Based on the target network model, feature extraction processing is performed on the target image information to obtain the target transformation matrix; Point cloud data is constructed based on the target transformation matrix; The step of performing feature extraction processing on target image information based on the target network model includes: performing global attention calculation on the target image information based on the target global attention module in the target network model to complete the feature extraction processing; The step of performing global attention calculation on the target image information based on the target global attention module in the target network model includes: obtaining attention weights through a first attention module in the target global attention module; multiplying the attention weights by the original input features to obtain global context features; obtaining the channel dependency relationship corresponding to the global context features according to a second attention module in the target global attention module; wherein the second attention module includes a bottleneck transformation module, a normalization layer, and an activation layer; and performing feature fusion processing with the target image information according to the channel dependency relationship to complete the global attention calculation.

2. The method according to claim 1, characterized in that, The global attention module is the Global Context Attention (GC) module.

3. The method according to claim 2, characterized in that, The first sub-network is a pose regression sub-network; the feature extraction module of the pose regression sub-network is a ResNet network.

4. The method according to claim 2, characterized in that, The second sub-network is a depth prediction sub-network; the depth prediction sub-network is a U-Net network; the U-Net network includes an encoder and a decoder; the encoder is a ResNet network; the decoder is a DispNet network.

5. The method according to claim 1, characterized in that, The step of performing self-supervised training based on the training dataset and the initial network model to obtain the target network model includes: A depth map and a transformation matrix are obtained based on adjacent frame images in the training dataset and the initial network model; wherein, the depth map represents the depth information of the previous frame image in the adjacent frame images; and the transformation matrix represents the relative transformation relationship between the adjacent frame images. The predicted image corresponding to the next frame in adjacent frame images is obtained based on the depth map and the transformation matrix. The loss function value is calculated based on the predicted image. The initial network model is then subjected to self-supervised training based on the loss function value to obtain the target network model.

6. The method according to claim 5, characterized in that, The step of obtaining the predicted image corresponding to the next frame image in adjacent frame images based on the depth map and the transformation matrix includes: The predicted image is obtained by performing an affine transformation based on the camera intrinsic parameters, the depth map, and the transformation matrix.

7. The method according to claim 5 or 6, characterized in that, The calculation of the loss function value based on the predicted image includes: A first loss function value is calculated based on the predicted image and the next frame image in the adjacent frame images; wherein, the first loss function value characterizes the degree of difference between the next frame image in the adjacent frame images and the predicted image; A second loss function value is calculated based on the depth map; wherein, the second loss function value characterizes the degree of change in the depth map; The loss function value is determined based on the first loss function value and the second loss function value.

8. The method according to claim 7, characterized in that, The step of calculating the first loss function value based on the predicted image and the next frame image in the adjacent frame images includes: The first loss function value is calculated based on the predicted image, the next frame image in the adjacent frame images, and the structural similarity index SSIM.

9. The method according to claim 5, characterized in that, The process of obtaining the depth map and transformation matrix based on adjacent frame images in the training dataset and the initial network model includes: Based on the first sub-network, feature extraction processing is performed on the adjacent frame images to obtain the transformation matrix; The depth map is obtained by performing feature extraction processing on the previous frame image based on the second sub-network.

10. The method according to claim 3 or 4, characterized in that, The ResNet network includes an input section, an output section, and an intermediate section; wherein the intermediate section is composed of stacked residual blocks.

11. The method according to claim 1, characterized in that, The step of performing feature extraction processing on the target image information based on the target network model to obtain the target transformation matrix includes: Based on the first target sub-network in the target network model, feature extraction processing is performed on the target image information to obtain the initial transformation matrix; Based on the target second sub-network in the target network model, feature extraction processing is performed on the previous frame image in the target image information to obtain the target depth map corresponding to the previous frame image in the target image information; The initial transformation matrix is ​​optimized using the target depth map to obtain the target transformation matrix.

12. A terminal, characterized in that, The terminal includes an acquisition unit, a training unit, an acquisition unit, and a construction unit. The acquisition unit is used to acquire the training dataset; The training unit is used to perform self-supervised training based on the training dataset and the initial network model to obtain the target network model; wherein, the initial network model includes a global attention module; the global attention module is used to learn the importance of different image patches and the importance of different channels; the global attention module includes a first attention module and a second attention module; the initial network model includes a first sub-network and a second sub-network; the first sub-network includes the first attention module; the second sub-network includes the second attention module; The acquisition unit is used to perform feature extraction processing on target image information based on the target network model to obtain a target transformation matrix; the acquisition unit is also used to perform global attention calculation on the target image information based on the target global attention module in the target network model to complete the feature extraction processing; the acquisition unit is also used to obtain attention weights through the first attention module in the target global attention module, multiply the attention weights by the original input features to obtain global context features; obtain the channel dependency relationship corresponding to the global context features according to the second attention module in the target global attention module; wherein, the second attention module includes a bottleneck transformation module, a normalization layer and an activation layer; perform feature fusion processing with the target image information according to the channel dependency relationship to complete the global attention calculation; The construction unit is used to construct point cloud data based on the target transformation matrix.

13. A terminal, characterized in that, The terminal includes a processor and a memory storing processor-executable instructions, which, when executed by the processor, implement the method as described in any one of claims 1-11.

14. A computer storage medium having a program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-11.

Images