Monocular vision simultaneous localization and mapping method based on digital twin data and semantic information

Abstract

The application provides a monocular vision instant positioning method based on digital twin data and semantic information, which has the following characteristics: S1, constructing a training data set according to real visual data collected in a preset area, virtual visual data collected by a virtual camera in a digital twin model corresponding to the preset area, and corresponding labels; S2, constructing a positioning model and a discriminator, and performing adversarial training on the positioning model according to the training data set and the discriminator to obtain a trained positioning model; and S3, inputting image data into the trained positioning model to obtain a positioning result. In summary, the method can generate accurate positioning results.

Description

Technical Field

[0001] This invention belongs to the field of visual positioning technology, specifically relating to a monocular visual real-time positioning method based on digital twin data and semantic information. Background Technology

[0002] Positioning technology refers to the technology or service of obtaining and labeling the location information of a carrier through specific methods. The Global Positioning System (GPS) is the most common positioning technology, but in certain scenarios, such as bridges, culverts, tunnels, complex indoor environments, and underground spaces where GPS signals are poor, related technologies struggle to provide continuous, accurate, and timely positioning results.

[0003] Simultaneous localization and mapping (SLAM), a visual information-based real-time localization technology, offers an effective alternative. SLAM and similar technologies leverage the widespread availability of cameras on smart devices to estimate the camera's pose in three-dimensional space, enabling real-time and accurate localization and navigation of mobile devices. This makes them suitable for applications such as virtual reality and robot navigation.

[0004] However, traditional visual information localization techniques often rely on dense frame monocular, binocular, and multi-view photography, and solve camera pose based on geometric constraints. This results in real-time localization drift due to the cumulative error of motion estimation. Furthermore, personal mobile devices and small drones have limited computing and storage resources, making it difficult to handle large-scale image data processing.

[0005] Therefore, there is an urgent need for a high-precision positioning method based on sparse image data. Summary of the Invention

[0006] This invention is made to solve the above-mentioned problems, and aims to provide a monocular visual real-time localization method based on digital twin data and semantic information.

[0007] This invention provides a monocular vision real-time localization method based on digital twin data and semantic information. It is used to obtain the localization result of a mobile device based on image data acquired by a monocular camera device installed on the mobile device. The method includes the following steps: Step S1, constructing a training dataset based on real visual data acquired in a preset area, and virtual visual data and corresponding labels acquired by a virtual camera in the corresponding digital twin model of the preset area; Step S2, constructing a localization model and a discriminator, and performing adversarial training on the localization model based on the training dataset and the discriminator to obtain a trained localization model; Step S3, inputting image data into the trained localization model to obtain the localization result. The localization model includes an image feature extraction module. The system includes an image encoder for extracting features from image data to obtain multiple image features of different sizes, and performing self-attention calculation on specified image features to obtain representative features; a semantic feature extraction module, including a text encoder and preset prompt words, for extracting features from prompt words through the text encoder to obtain semantic features; a similarity scoring table generation module, for calculating similarity based on semantic features and representative features to obtain a similarity scoring table; a residual connection module, for extracting global information from image data to obtain global features; a decoding module, for obtaining a decoded image based on global features, the similarity scoring table, and all image features; and a localization module, including a regression head, for generating pose information of the monocular camera device as the localization result based on the decoded image.

[0008] The monocular vision real-time localization method based on digital twin data and semantic information provided by this invention may also have the following features: the residual connection module includes: a patch embedding unit, used to adjust the data size of the image data according to the maximum size corresponding to the image features; and a large kernel attention unit, used to extract global information from the adjusted image data to obtain global features.

[0009] The monocular visual real-time localization method based on digital twin data and semantic information provided by this invention may also have the following features: wherein the large-kernel attention unit includes a first batch of normalization layer, attention layer, first connection layer, second batch of normalization layer, feedforward neural network and second connection layer connected in sequence. The input of the first batch of normalization layer is the adjusted image data, the input of the first connection layer is the output of the attention layer and the adjusted image data, the input of the second connection layer is the output of the second batch of normalization layer and the feedforward neural network, and the output is global features.

[0010] The monocular visual real-time localization method based on digital twin data and semantic information provided by the present invention may also have the following feature: wherein the attention layer includes a first convolutional sub-layer, a first GELU activation function sub-layer, an MLKA sub-layer, and a second convolutional sub-layer connected in sequence.

[0011] The monocular visual real-time localization method based on digital twin data and semantic information provided by this invention may also have the following feature: wherein the feedforward neural network includes a third convolutional sub-layer, a fourth convolutional sub-layer, a second GELU activation function sub-layer, and a fifth convolutional layer connected in sequence.

[0012] The monocular vision real-time localization method based on digital twin data and semantic information provided by this invention may also have the following features: In the decoding module, global features are sequentially superimposed with image features of decreasing size, and each superimposed feature is convolved to compress the number of channels of the superimposed feature to a preset number of channels. The superimposed feature with the preset number of channels is used as a new global feature and superimposed with the next feature image. Before superposition, all image features are compressed to the preset number of channels through convolution. The superimposed feature with the preset number of channels corresponding to the last image feature is convolved with the similarity scoring table to obtain the decoded image.

[0013] The monocular visual instantaneous localization method based on digital twin data and semantic information provided by this invention may also have the following feature: wherein the specified image feature is the smallest image feature among all image features.

[0014] The monocular visual instantaneous localization method based on digital twin data and semantic information provided by this invention may also have the following features: the training dataset includes a subset of real data constructed from real visual data and a subset of synthetic data constructed from virtual visual data and corresponding labels. In step S2, the localization model is subjected to adversarial training based on the synthetic data subset and tested based on the real data subset until the error on the real data subset is minimized, thus obtaining the trained localization model.

[0015] The monocular visual real-time localization method based on digital twin data and semantic information provided by this invention may also have the following feature: wherein the loss function for adversarial training includes regression loss. and classification loss The formula for calculating regression loss is: In the formula, x represents the coordinates of the virtual camera in the pose information predicted by the localization model. q represents the real coordinates of the virtual camera in the label, and q represents the azimuth angle of the virtual camera in the pose information predicted by the localization model. The true azimuth angle of the virtual camera in the label is γ, where γ is the hyperparameter error ratio and the classification loss is 0. The calculation expression is: In the formula, D is the discriminator, and G(I) t The localization model is based on the input image data I.t The generated decoded image.

[0016] The role and effect of invention

[0017] According to the monocular vision real-time localization method based on digital twin data and semantic information of this invention, on the one hand, a large amount of training data is acquired through the digital twin model, and the localization model is trained by combining it with training data from the real environment, thereby improving the predictive performance of the localization model; on the other hand, the expressive power of image features is enhanced through the residual connection module, and the outputs of the residual connection module, the image feature extraction module, and the similarity scoring table generation module are fused using a decoder, thereby obtaining accurate localization results using a regression head. Therefore, the monocular vision real-time localization method based on digital twin data and semantic information of this invention can generate accurate localization results. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the monocular visual real-time localization method based on digital twin data and semantic information in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the training dataset in an embodiment of the present invention.

[0020] Figure 3 This is a block diagram of the positioning model in an embodiment of the present invention;

[0021] Figure 4 This is a block diagram of the large-core attention unit in an embodiment of the present invention. Detailed Implementation

[0022] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the monocular visual real-time localization method based on digital twin data and semantic information of this invention.

[0023] This embodiment provides a monocular vision real-time localization method based on digital twin data and semantic information, which is used to obtain the localization result of the mobile device based on image data collected by a monocular camera device installed on the mobile device.

[0024] Figure 1 This is a flowchart illustrating the monocular visual real-time localization method based on digital twin data and semantic information in an embodiment of the present invention.

[0025] like Figure 1 As shown, the monocular visual real-time localization method based on digital twin data and semantic information includes the following steps:

[0026] Step S1: Construct a training dataset based on the real visual data collected in the preset area, and the virtual visual data and corresponding labels collected by the virtual camera in the digital twin model corresponding to the preset area.

[0027] Figure 2 This is a schematic diagram of the training dataset in an embodiment of the present invention.

[0028] like Figure 2 As shown, for a real, preset region, a subset of real data is constructed by collecting real visual data using a camera device. A digital twin model is then built for the preset region, and a synthetic data subset is constructed by collecting virtual visual data and corresponding pose parameters using a virtual camera with known pose parameters as labels. Therefore, the final training dataset includes both the real data subset and the synthetic data subset.

[0029] In this embodiment, the real visual data can be fixed- or non-fixed-interval image samples or video streams. The sampling frequency is unlimited, and multiple images do not need to contain the same observed object. In this embodiment, the virtual visual data is fixed- or non-fixed-interval image samples or video streams collected along a certain trajectory in a digital twin model.

[0030] Step S2: Construct a localization model and a discriminator, and perform adversarial training on the localization model based on the training dataset and the discriminator to obtain a trained localization model.

[0031] Figure 3 This is a block diagram of the positioning model in an embodiment of the present invention.

[0032] like Figure 3 As shown, the localization model 100 includes an image feature extraction module 11, a semantic feature extraction module 12, a similarity scoring table generation module 13, a residual connection module 14, a decoding module 15, and a localization module 16.

[0033] The image feature extraction module 11 includes an image encoder, which is used to extract features from image data to obtain multiple image features of different sizes, and to perform self-attention calculation on specified image features to obtain representative features.

[0034] The specified image feature is the smallest image feature among all image features. In this embodiment, four image features of different sizes are obtained and sorted from largest to smallest as image feature A1, image feature A2, image feature A3, and image feature A4. The specified image feature is image feature A4. In other embodiments, an existing image encoder can be used without modification. In this case, for multiple output image features, an image feature of a specific size, rather than the smallest size, can be selected as the specified image feature.

[0035] In this embodiment, the calculation expression representing the feature is:

[0036]

[0037] In the formula For representative features, MHSA is a self-attention computation, and x4 represents the specified image features. The mean of the specified image features.

[0038] The semantic feature extraction module 12 includes a text encoder and preset prompt words, which are used to extract features from the prompt words through the text encoder to obtain semantic features.

[0039] In this embodiment, the text encoder of the semantic feature extraction module 12 and the image encoder of the image feature extraction module 11 are respectively the text encoder and image encoder of the existing contrastive language-image pre-trained model, namely the CLIP model. In other embodiments, the image encoder and text encoder of other multimodal models can be used as the text encoder of the semantic feature extraction module 12 and the image encoder of the image feature extraction module 11.

[0040] In this embodiment, the preset prompt words are obtained sequentially through a prompt word pre-selection step and a prompt word setting step. The prompt word pre-selection step involves inputting training data from a subset of the synthetic data into the localization model 100, along with a list of candidate prompt words containing multiple candidate words. After the localization model 100 converges based on the training data, the candidate prompt words are pre-selected according to their corresponding weights in the candidate prompt word list, resulting in a set of multiple pre-selected prompt words. The prompt word setting step involves manually selecting suitable prompt words from the pre-selected prompt word set as preset prompt words, or directly selecting suitable pre-selected prompt words based on the characteristics of the digital twin model.

[0041] The similarity scoring table generation module 13 is used to calculate similarity based on semantic features and representative features to obtain a similarity scoring table.

[0042] The residual connection module 14 is used to extract global information from the image data to obtain global features.

[0043] The residual connection module 14 includes a patch embedding unit 141 and a large kernel attention unit 142.

[0044] The patch embedding unit 141 is used to adjust the data size of the image data according to the maximum size corresponding to the image features.

[0045] The large kernel attention unit 142 is used to extract global information from the adjusted image data to obtain global features.

[0046] Figure 4 This is a block diagram of the large-core attention unit in an embodiment of the present invention.

[0047] like Figure 4 As shown, the large kernel attention unit 142 includes a first batch of normalization layer 1421, attention layer 1422, first connection layer 1423, second batch of normalization layer 1424, feedforward neural network 1425, and second connection layer 1426 connected in sequence.

[0048] The input to the first normalization layer 1421 is the adjusted image data. In this embodiment, the first normalization layer 1421 and the second normalization layer 1424 perform batch normalization on the input, making its mean 0 and variance 1.

[0049] The attention layer 1422 includes a first convolutional sub-layer 14221, a first GELU activation function sub-layer 14222, an MLKA sub-layer 14223, and a second convolutional sub-layer 14224 connected in sequence.

[0050] The input to the first connection layer 1423 is the output of the attention layer 1422 and the adjusted image data.

[0051] The feedforward neural network 1425 includes a third convolutional sub-layer 14251, a fourth convolutional sub-layer 14252, a second GELU activation function sub-layer 14253, and a fifth convolutional sub-layer 14254 connected in sequence.

[0052] The number of channels in the fourth convolutional sublayer 14252 is a preset number of channels, which is 256 in this embodiment.

[0053] The input to the second connection layer 1426 is the output of the second batch normalization layer 1424 and the feedforward neural network 1425, and the output is the global features.

[0054] The decoding module 15 includes multiple convolutional layers to obtain the decoded image based on global features, a similarity scoring table, and all image features.

[0055] In the decoding module 15, the global feature is sequentially superimposed with image features of decreasing size, and each superimposed feature is convolved to compress its channel count to a preset number. The superimposed feature with the preset number of channels is then used as a new global feature and superimposed with the next feature image. Before superposition, all image features are compressed to the preset number of channels through convolution. The superimposed feature with the preset number of channels corresponding to the last image feature is convolved with the similarity scoring table to obtain the decoded image.

[0056] For example, in this embodiment, image features A1, A2, A3, and A4 are all compressed to a preset number of channels using 1x1 convolution. Then, image feature A4 and the global feature are sequentially superimposed and compressed to the preset number of channels using convolution to obtain a new global feature. The global feature is then sequentially superimposed and compressed with image features A3, A2, and A1 as described above. Finally, the obtained global feature is convolved with a similarity scoring table to obtain the decoded image.

[0057] The positioning module 16 includes a regression head, which is used to generate pose information of the monocular camera device as a positioning result based on the decoded image.

[0058] In this embodiment, the localization model 100 undergoes adversarial training based on a synthetic data subset and is tested on a real data subset until the error on the real data subset is minimized, thus obtaining a trained localization model. In this embodiment, the error corresponding to the synthetic data subset is less than a preset threshold, and the domain difference error is minimized through adversarial training using a discriminator, thereby simultaneously minimizing the error on the real data subset. In this embodiment, the parameters of the localization model 100 and the discriminator are sequentially and iteratively optimized through adversarial training, wherein the parameters of the residual connection module 14 and the decoding module 15 of the localization model 100 are optimized.

[0059] The loss function for adversarial training includes regression loss. and classification loss

[0060] The formula for calculating regression loss is:

[0061]

[0062]

[0063] In the formula, x represents the coordinates of the virtual camera in the pose information predicted by the localization model. q represents the real coordinates of the virtual camera in the label, and q represents the azimuth angle of the virtual camera in the pose information predicted by the localization model. γ represents the true azimuth angle of the virtual camera in the label, and γ is the hyperparameter error ratio.

[0064] Classification loss The calculation expression is:

[0065]

[0066] In the formula, D is the discriminator, and G(I) t The localization model is based on the input image data I. t The generated decoded image.

[0067] Step S3: Input the image data into the trained localization model 100 to obtain the localization result. In this embodiment, the image data input into the trained localization model 100 is a single image. In other embodiments, by adding a corresponding module for time series processing to the localization model 100 and setting the corresponding training loss, the input of the model can be converted into multiple images.

[0068] The role and effect of the embodiments

[0069] According to the monocular vision real-time localization method based on digital twin data and semantic information involved in this embodiment, on the one hand, a large amount of training data is obtained through the digital twin model, and the localization model is trained by combining it with training data from the real environment to improve the predictive performance of the localization model; on the other hand, the expressive power of image features is enhanced through the residual connection module, and the outputs of the residual connection module, the image feature extraction module, and the similarity scoring table generation module are fused using a decoder, and then an accurate localization result is obtained using a regression head. In summary, this method does not rely on the results of prior estimation and will not experience cumulative drift during localization, thus generating accurate localization results.

[0070] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A monocular vision real-time localization method based on digital twin data and semantic information, used to obtain the localization result of a mobile device based on image data acquired by a monocular camera device installed on the mobile device, characterized in that, Includes the following steps: Step S1: Construct a training dataset based on real visual data collected in a preset area, virtual visual data collected by a virtual camera in the digital twin model corresponding to the preset area, and the corresponding labels. The virtual visual data collected by a virtual camera with known pose parameters and the corresponding pose parameters are used as the labels. Step S2: Construct a localization model and a discriminator, and perform adversarial training on the localization model based on the training dataset and the discriminator to obtain a trained localization model; Step S3: Input the image data into the trained localization model to obtain the localization result. The positioning model includes: The image feature extraction module includes an image encoder, which is used to extract features from the image data to obtain multiple image features of different sizes, and to perform self-attention calculation on the specified image features to obtain representative features; The semantic feature extraction module includes a text encoder and preset prompt words, used to extract features from the prompt words through the text encoder to obtain semantic features; The similarity scoring table generation module is used to calculate the similarity based on the semantic features and the representative features to obtain a similarity scoring table; The residual connection module is used to extract global information from the image data to obtain global features; The decoding module is used to obtain a decoded image based on the global features, the similarity scoring table, and all the image features; The positioning module includes a regression head, used to generate pose information of the monocular camera device as the positioning result based on the decoded image.

2. The monocular visual real-time localization method based on digital twin data and semantic information according to claim 1, Its features are: The residual connection module includes: A patch embedding unit is used to adjust the data size of the image data according to the maximum size corresponding to the image feature; The large kernel attention unit is used to extract global information from the adjusted image data to obtain the global features.

3. The monocular visual real-time localization method based on digital twin data and semantic information according to claim 2, characterized in that: in, The large-kernel attention unit comprises, in sequence, a first batch of normalization layers, an attention layer, a first connection layer, a second batch of normalization layers, a feedforward neural network, and a second connection layer. The input to the first batch of normalization layers is the adjusted image data. The input to the first connection layer is the output of the attention layer and the adjusted image data. The input to the second connection layer is the output of the second batch normalization layer and the feedforward neural network, and the output is the global feature.

4. The monocular visual real-time localization method based on digital twin data and semantic information according to claim 3, characterized in that: in, The attention layer comprises a first convolutional sub-layer, a first GELU activation function sub-layer, an MLKA sub-layer, and a second convolutional layer connected in sequence.

5. The monocular visual real-time localization method based on digital twin data and semantic information according to claim 3, characterized in that: in, The feedforward neural network includes a third convolutional sub-layer, a fourth convolutional sub-layer, a second GELU activation function sub-layer, and a fifth convolutional sub-layer connected in sequence.

6. The monocular visual real-time localization method based on digital twin data and semantic information according to claim 1, characterized in that: in, In the decoding module, the global feature is sequentially superimposed with each of the image features in descending order of size, and convolution processing is performed on each superimposed feature to compress the number of channels of the superimposed feature to a preset number of channels. The features obtained by superimposing the preset number of channels are used as the new global features and superimposed on the next image feature. Before stacking, all the image features are compressed to the preset number of channels through convolution processing. The last image feature, corresponding to the superimposed feature with the preset number of channels, is convolved with the similarity scoring table to obtain the decoded image.

7. The monocular visual real-time localization method based on digital twin data and semantic information according to claim 1, characterized in that: in, The specified image feature is the image feature with the smallest size among all the image features.

8. The monocular visual real-time localization method based on digital twin data and semantic information according to claim 1, characterized in that: in, The training dataset includes a subset of real data constructed from the real visual data and a subset of synthetic data constructed from the virtual visual data and corresponding labels. In step S2, the localization model is trained adversarially based on the synthetic data subset and tested based on the real data subset until the error on the real data subset is minimized, thus obtaining the trained localization model.

9. The monocular visual real-time localization method based on digital twin data and semantic information according to claim 8, characterized in that: in, The loss function for adversarial training includes regression loss. and classification loss , The formula for calculating the regression loss is: ， ， ， In the formula The coordinates of the virtual camera in the pose information predicted by the localization model. The actual coordinates of the virtual camera mentioned in the label. The azimuth angle of the virtual camera in the pose information predicted by the positioning model. The actual azimuth angle of the virtual camera mentioned in the label. This represents the proportion of hyperparameter error. The classification loss The calculation expression is: ， In the formula For the discriminator, The localization model is based on the input image data. The generated decoded image.

Images