BIM-assisted indoor positioning method and system based on cross-domain image retrieval

Abstract

The invention belongs to the technical field of indoor positioning, and provides a BIM-assisted indoor positioning method and system based on cross-domain image retrieval, and the method comprises the steps: building a BIM model, capturing a roaming image, obtaining a test image, synthesizing the image, retrieving a most similar synthesized image, mapping the image, and extracting a position. According to the method, the problem of motion blur in the image retrieval positioning process is solved through the NAFNet network; through the improved CycleGAN network, high-similarity synthesis from a virtual acquisition image to a real image is realized, the authenticity and credibility of the synthesized image are improved, and the cross-domain difference between a BIM rendering image and a real indoor photo is reduced; the feature extraction and the efficient retrieval of the image are realized through the SwinTransform; through the SRPnP algorithm, the accuracy and speed of indoor positioning are improved.

Description

Technical Field

[0001] The present invention relates to the field of indoor positioning technology, and in particular to a BIM-assisted indoor positioning method and system based on cross-domain image retrieval. Background Art

[0002] Indoor camera pose estimation is the key to realizing location-based services in built environments. Economical, efficient, and accurate indoor positioning technology is of great significance to the advancement of modernization. Image retrieval methods are used to obtain location information, mainly by selecting the best matching image, retrieving the nearest neighbor image similar to the query image in the image database, and calculating the camera pose using the 3D model information of the nearest neighbor image. Among these methods, an analytical solution based on photogrammetry can be used to estimate the camera pose with six degrees of freedom. However, image retrieval methods and their variations require a large database of photos with known locations or a pre-existing map of the entire indoor space, which is highly labor-intensive and time-consuming to obtain.

[0003] Building Information Modeling, or BIM for short, provides a three-dimensional spatial model of built facilities, which contains the visual appearance and spatial layout of the indoor environment. However, there is a cross-domain gap in reality. Given a BIM-rendered image and a photo of the same indoor scene, the computer cannot detect the correct scale-invariant feature transformation correspondence. This phenomenon suggests that the computer perceives BIM-rendered images and real-life photographs in different ways, which makes it difficult to directly perform image retrieval or similarity comparison based on registration of SIFT, HOG or other image descriptors. In addition, how to automatically and quickly obtain BIM image datasets through software such as Revit and Blender to prepare for subsequent work is also an unresolved problem. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a BIM-assisted indoor positioning method and system based on cross-domain image retrieval, solve the motion blur problem in the image retrieval and positioning process through the NAFNet network; realize high similarity synthesis of virtual captured images to real images through the improved CycleGAN network, improve the authenticity and credibility of the synthesized images, and reduce the cross-domain differences between BIM rendered images and real indoor photos; realize feature extraction and efficient retrieval of images through Swin Transformer; and improve the accuracy and speed of indoor positioning through the SRPnP algorithm.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A BIM-assisted indoor positioning method based on cross-domain image retrieval, comprising:

[0007] Build BIM models;

[0008] Using a virtual camera in the Revit software to roam the BIM model, and capturing roaming images at a predefined frame rate to obtain a BIM rendering image dataset;

[0009] Taking an indoor image of a target building, and removing dynamic blur from the indoor image of the target building using a NAFNet network to obtain a test image;

[0010] Using the trained improved CycleGAN network to convert the BIM rendering image dataset into a synthetic image set;

[0011] Using a pre-trained Swin Transformer model, performing feature extraction, cosine similarity calculation, and image retrieval on the test image and the synthetic image set to obtain the most similar synthetic image;

[0012] Using the SIFT algorithm to map the most similar synthetic image from two dimensions to three dimensions to obtain mapping relationship data;

[0013] The SRPnP algorithm is used to estimate the six-degree-of-freedom posture of the camera on the mapping relationship data to obtain an indoor position recognition result.

[0014] Preferably, the BIM model is roamed using a virtual camera in the Revit software, and roaming images are captured at a predefined frame rate to obtain a BIM rendering image dataset, including:

[0015] Configure the generated property options of the Revit software; the property options include: wireframe, hatching, color options, and rendering options;

[0016] Setting a roaming path of the virtual camera; the roaming path includes: a frame interval and a predetermined walking frame;

[0017] Based on the Revit software configured with the generation property option, the virtual camera is used to collect images of the BIM model according to the roaming path to obtain the BIM rendering image dataset in XML format

[0018] Preferably, shooting an indoor image of a target building, and removing dynamic blur from the indoor image of the target building using a NAFNet network to obtain a test image includes:

[0019] Using a camera to photograph the interior of a target building to obtain an indoor image of the target building;

[0020] The image fuzzy discrimination metric and fuzzy discrimination threshold of the target building indoor image are calculated; the calculation formula of the image fuzzy discrimination metric is:

[0021]

[0022] The calculation formula of the fuzzy discrimination threshold is:

[0023]

[0024] Wherein, BDM is the image blur discrimination metric; T is the blur discrimination threshold; H is the image height; W is the image width; g x (i, j) is the gradient of the image in the x direction; g y (i, j) is the gradient of the image in the y direction; N is the total number of gradient values ​​calculated for the image in the x and y directions; G n is the sum of non-zero gradient values; n is the number of scenes; and are respectively the maximum and minimum values ​​of the image blur discrimination metric;

[0025] Performing blur judgment on the indoor image of the target building according to the image blur discrimination metric and the blur discrimination threshold to obtain blur degree data;

[0026] When the blur degree data exceeds a preset blur allowable range, the NAFNet network is used to perform dynamic blur removal on the indoor image of the target building to obtain the test image.

[0027] Preferably, the training process of the improved CycleGAN network includes:

[0028] The CBDM is integrated into the residual block of the generator of the basic CycleGAN network, the ReLU activation function in the convolutional layer of the generator of the basic CycleGAN network is replaced by the PReLU activation function, and the label smoothing technique is used to optimize the discriminant module of the basic CycleGAN network to obtain an improved CycleGAN network;

[0029] The pre-collected BIM rendering images are used as the training set, and the pre-collected indoor shooting images are used as the validation set;

[0030] Using the generator in the improved CycleGAN network to convert the training set into an indoor shooting style, to obtain a generated image set;

[0031] Using the discriminator in the improved CycleGAN network to judge the authenticity of the generated image set, and obtain a judgment result;

[0032] When the judgment result is generation failure, the loss value of the generated image set is calculated to obtain an intermediate adversarial loss, and the parameters of the improved CycleGAN network are optimized and iterated using the intermediate adversarial loss to obtain the improved CycleGAN network after iteration. The calculation formula of the adversarial loss is:

[0033]

[0034] in, is the adversarial loss; A and B are both source domains; G AB and G BA are all the generators mentioned above; D B is the discriminator; G BA (G AB (a)) is an image in the generated image set; is the real adversarial loss; To generate adversarial loss;

[0035] The generation similarity of the iteratively completed improved CycleGAN network is calculated to obtain the Frechette starting distance; the calculation formula of the Frechette starting distance is:

[0036] FID=||μ r -μ g || 2 +Tr(∑r+∑g-2(∑r∑g) 1 / 2 );

[0037] Wherein, FID is the Frechette starting distance; μ r , μ g are the feature means of the validation set and the generated image set respectively; ∑r and ∑g are the covariance matrices of the validation set and the generated image set respectively; Tr is the trace;

[0038] When the Frechette starting distance meets the preset similarity standard, the trained improved CycleGAN network is output.

[0039] Preferably, the pre-trained Swin Transformer model is used to perform feature extraction, cosine similarity calculation and image retrieval on the test image and the synthetic image set to obtain the most similar synthetic image, including:

[0040] Using the Swin Transformer model, feature extraction is performed on the test image and the synthetic image set to obtain a test feature and a synthetic feature set;

[0041] Calculating the cosine similarity between the test feature and each synthetic feature in the synthetic feature set to obtain a similarity data set;

[0042] According to the similarity data set, a synthetic image with the largest similarity value in the synthetic image set is determined as the most similar synthetic image.

[0043] Preferably, the most similar composite image is mapped from two dimensions to three dimensions using a SIFT algorithm to obtain mapping relationship data, including:

[0044] Using SIFT algorithm to obtain local feature points of the most similar composite image;

[0045] The three-dimensional world coordinates of the local feature points are obtained by using the UIDoc.Selection.PickObject function in the Revit software;

[0046] According to the most similar synthetic image, pixel mapping is performed using the three-dimensional world coordinates to obtain the mapping relationship data.

[0047] Preferably, the calculation formula of the indoor location recognition result is: i ·p i =K[R|t]P i ; Where R is the rotation matrix; t is the translation vector; s i is the unknown proportional factor; p i is the i-th point on the most similar composite image; P i are the homogeneous coordinates of the i-th three-dimensional point; K is the internal reference matrix of the virtual camera; [R|t] is the transformation from the world coordinate system to the camera coordinate system.

[0048] Preferably, a BIM-assisted indoor positioning system based on cross-domain image retrieval includes:

[0049] Model building module, used to build BIM models;

[0050] A roaming image capture module is used to roam the BIM model using a virtual camera in the Revit software, and to capture roaming images at a predefined frame rate to obtain a BIM rendering image data set;

[0051] A test image acquisition module is used to capture an indoor image of a target building, and use a NAFNet network to remove dynamic blur from the indoor image of the target building to obtain a test image;

[0052] An image synthesis module, used to convert the BIM rendering image dataset into a synthetic image set using the trained improved CycleGAN network;

[0053] A most similar synthetic image retrieval module, used to perform feature extraction, cosine similarity calculation and image retrieval on the test image and the synthetic image set using a pre-trained Swin Transformer model to obtain the most similar synthetic image;

[0054] A mapping module, used to map the most similar synthetic image from two dimensions to three dimensions using a SIFT algorithm to obtain mapping relationship data;

[0055] The position extraction module is used to use the SRPnP algorithm to perform camera six-degree-of-freedom posture estimation on the mapping relationship data to obtain indoor position recognition results.

[0056] The present invention discloses the following technical effects:

[0057] The present invention provides a BIM-assisted indoor positioning method and system based on cross-domain image retrieval. The NAFNet network is used to solve the problem of motion blur in the image retrieval and positioning process, and the optimization of the image to be identified is realized. The improved CycleGAN network is used to solve the problem of poor synthesis effect of the existing CycleGAN network, and the high similarity synthesis of virtual collected images to real images is realized. The Swin Transformer is used to solve the defects of poor feature extraction effect and low retrieval efficiency of the existing retrieval mechanism, and the feature extraction and efficient retrieval of the image are realized. The SRPnP algorithm is used to solve the problem of poor anti-interference ability of the existing position extraction method, and the reliable recognition of indoor positions is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0059] Figure 1 A schematic diagram of a BIM-assisted indoor positioning process based on cross-domain image retrieval provided by an embodiment of the present invention;

[0060] Figure 2 A schematic diagram of the structure of a BIM-assisted indoor positioning model based on cross-domain image retrieval provided by an embodiment of the present invention;

[0061] Figure 3 A schematic diagram of the improved CycleGAN network structure provided by an embodiment of the present invention;

[0062] Figure 4 A schematic diagram of implementing the SRPnP algorithm provided in an embodiment of the present invention;

[0063] Figure 5 A schematic diagram of the hardware structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The purpose of the present invention is to provide a BIM-assisted indoor positioning method and system based on cross-domain image retrieval, solve the motion blur problem in the image retrieval and positioning process through the NAFNet network; realize high similarity synthesis of virtual captured images to real images through the improved CycleGAN network, improve the authenticity and credibility of the synthesized images, and reduce the cross-domain differences between BIM rendered images and real indoor photos; realize feature extraction and efficient retrieval of images through the Swin Transformer; and improve the accuracy and speed of indoor positioning through the SRPnP algorithm.

[0066] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] Figure 1 A schematic diagram of a BIM-assisted indoor positioning process based on cross-domain image retrieval provided by an embodiment of the present invention, Figure 2 A schematic diagram of the structure of a BIM-assisted indoor positioning model based on cross-domain image retrieval provided by an embodiment of the present invention is shown in FIG. Figure 1 and Figure 2 As shown, the present invention provides a BIM-assisted indoor positioning method based on cross-domain image retrieval, comprising:

[0068] Step 100: Build a BIM model;

[0069] Step 200: roaming the BIM model using a virtual camera in the Revit software, and capturing roaming images at a predefined frame rate to obtain a BIM rendering image dataset;

[0070] Step 300: photographing an indoor image of a target building, and using a NAFNet network to remove dynamic blur from the indoor image of the target building to obtain a test image;

[0071] Step 400: using the trained improved CycleGAN network to convert the BIM rendering image dataset into a synthetic image set;

[0072] Step 500: using a pre-trained Swin Transformer model to perform feature extraction, cosine similarity calculation, and image retrieval on the test image and the synthetic image set to obtain the most similar synthetic image;

[0073] Step 600: Mapping the most similar composite image from two dimensions to three dimensions using a SIFT algorithm to obtain mapping relationship data;

[0074] Step 700: Use the SRPnP algorithm to perform camera six-degree-of-freedom posture estimation on the mapping relationship data to obtain indoor position recognition results.

[0075] Specifically, we developed Revit to roam around the BIM environment space, automatically generated a BIM model rendering image dataset and location information annotations; and deblurred the captured query image. We improved CycleGAN by optimizing the generator and discriminator to achieve style conversion from BIM images to indoor captured images; we used the Swin Transformer network to extract features from the style-converted synthetic image and the query image, and calculated the similarity, returning the synthetic image with the highest similarity, and associated it with the corresponding BIM image. We constructed the corresponding relationship between the two-dimensional pixel coordinates of the retrieved image and the three-dimensional coordinates of the space through BIM, and used the SRPnP technology to estimate the camera pose, and obtained the 6-DOF relative position information to achieve indoor positioning.

[0076] Preferably, the BIM model is roamed using a virtual camera in the Revit software, and roaming images are captured at a predefined frame rate to obtain a BIM rendering image dataset, including:

[0077] Configure the generated property options of the Revit software; the property options include: wireframe, hatching, color options, and rendering options;

[0078] Setting a roaming path of the virtual camera; the roaming path includes: a frame interval and a predetermined walking frame;

[0079] Based on the Revit software configured with the generation property option, the virtual camera is used to collect images of the BIM model according to the roaming path to obtain the BIM rendering image dataset in XML format

[0080] Specifically, an indoor image of a target building is photographed, and dynamic blur removal is performed on the indoor image of the target building using a NAFNet network to obtain a test image, including:

[0081] Using a camera to photograph the interior of a target building to obtain an indoor image of the target building;

[0082] The image fuzzy discrimination metric and fuzzy discrimination threshold of the target building indoor image are calculated; the calculation formula of the image fuzzy discrimination metric is:

[0083]

[0084] The calculation formula of the fuzzy discrimination threshold is:

[0085]

[0086] Wherein, BDM is the image blur discrimination metric; T is the blur discrimination threshold; H is the image height; W is the image width; g x (i, j) is the gradient of the image in the x direction; g y (i, j) is the gradient of the image in the y direction; N is the total number of gradient values ​​calculated for the image in the x and y directions; G n is the sum of non-zero gradient values; n is the number of scenes; and are respectively the maximum and minimum values ​​of the image blur discrimination metric;

[0087] Performing blur judgment on the indoor image of the target building according to the image blur discrimination metric and the blur discrimination threshold to obtain blur degree data;

[0088] When the blur degree data exceeds a preset blur allowable range, the NAFNet network is used to perform dynamic blur removal on the indoor image of the target building to obtain the test image.

[0089] refer to Figure 3 , the training process of the improved CycleGAN network includes:

[0090] The CBDM is integrated into the residual block of the generator of the basic CycleGAN network, the ReLU activation function in the convolutional layer of the generator of the basic CycleGAN network is replaced by the PReLU activation function, and the label smoothing technique is used to optimize the discriminant module of the basic CycleGAN network to obtain an improved CycleGAN network;

[0091] The pre-collected BIM rendering images are used as the training set, and the pre-collected indoor shooting images are used as the validation set;

[0092] Using the generator in the improved CycleGAN network to convert the training set into an indoor shooting style, to obtain a generated image set;

[0093] Using the discriminator in the improved CycleGAN network to judge the authenticity of the generated image set, and obtain a judgment result;

[0094] When the judgment result is generation failure, the loss value of the generated image set is calculated to obtain an intermediate adversarial loss, and the parameters of the improved CycleGAN network are optimized and iterated using the intermediate adversarial loss to obtain the improved CycleGAN network after iteration. The calculation formula of the adversarial loss is:

[0095]

[0096] in, is the adversarial loss; A and B are both source domains; G AB and G BA are all the generators mentioned above; D B is the discriminator; G BA (G AB (a)) is an image in the generated image set; is the real adversarial loss; To generate adversarial loss;

[0097] The generation similarity of the iteratively completed improved CycleGAN network is calculated to obtain the Frechette starting distance; the calculation formula of the Frechette starting distance is:

[0098] FID=||μ r -μ g || 2 +Tr(∑r+∑g-2(∑r∑g) 1 / 2 );

[0099] Wherein, FID is the Frechette starting distance; μ r , μ g are the feature means of the validation set and the generated image set respectively; ∑r and ∑g are the covariance matrices of the validation set and the generated image set respectively; Tr is the trace;

[0100] When the Frechette starting distance meets the preset similarity standard, the trained improved CycleGAN network is output.

[0101] Specifically, the pre-trained Swin Transformer model is used to perform feature extraction, cosine similarity calculation and image retrieval on the test image and the synthetic image set to obtain the most similar synthetic image, including:

[0102] Using the Swin Transformer model, feature extraction is performed on the test image and the synthetic image set to obtain a test feature and a synthetic feature set;

[0103] Calculating the cosine similarity between the test feature and each synthetic feature in the synthetic feature set to obtain a similarity data set;

[0104] According to the similarity data set, a synthetic image with the largest similarity value in the synthetic image set is determined as the most similar synthetic image.

[0105] Specifically, the CycleGAN adversarial generative network is improved to convert BIM rendering images into images in the style of real photography. The model generator and discriminator are optimized according to the actual situation to improve the image style conversion effect. The detailed improvements are as follows:

[0106] 1) CBAM is added to the residual block of the CycleGAN generator, so that the generator can focus on the key parts of the input image more accurately; CBAM improves the performance of the generator and the quality of the generated image by learning the importance of each channel and the relationship between channels, and solves the problem of structural deformity encountered by the generator network when processing complex images, thereby improving the performance and reliability of the generation model;

[0107] 2) In the original CycleGAN model, the generator convolution layer uses the ReLU activation function, which is sensitive to the learning rate. A training model with a smaller learning rate may fail to converge or fall into a local extreme value. The ReLU activation function in the generator convolution layer is replaced with the PReLU activation function. ReLU may have an output of 0 during training, resulting in the inability to update the gradient. The PReLU activation function is an improved version of the ReLU activation function, which introduces learnable parameters to solve the problem of constant output caused by the ReLU slope of 0 when negative input is input. Compared with the ReLU activation function, the PReLU activation function can more flexibly adjust the activation degree of negative input, improve the nonlinear modeling ability of the generator, better learn complex mapping relationships, capture the differences between the two fields, generate more realistic and detailed real images, and perform image conversion more accurately.

[0108] 3) The original label is in One-Hot hard label format, that is, the labels of the original judgment samples are 1 and 0 respectively when they are true or false; label smoothing is used in the discriminator network to convert the hard label into a soft label through a tiny smoothing operation, that is, if the judgment is true, a random number from 0.8 to 1.2 is output instead of the original 1, and if the judgment is false, a random number from 0 to 0.3 is output instead of the original 0; the problem of absolute confidence of the model is alleviated, the robustness of the model is improved, the lack of supervision information in the binary classification is improved, the richness of the data is increased, and the adaptability of the model is enhanced.

[0109] refer to Figure 4 , using the SIFT algorithm to map the most similar synthetic image from two dimensions to three dimensions, to obtain mapping relationship data, including:

[0110] Using SIFT algorithm to obtain local feature points of the most similar composite image;

[0111] The three-dimensional world coordinates of the local feature points are obtained by using the UIDoc.Selection.PickObject function in the Revit software;

[0112] According to the most similar synthetic image, pixel mapping is performed using the three-dimensional world coordinates to obtain the mapping relationship data.

[0113] Specifically, the calculation formula for the indoor location recognition result is: i ·p i =K[R|t]P i ; Where R is the rotation matrix; t is the translation vector; s i is the unknown proportional factor; p i is the i-th point on the most similar composite image; P i are the homogeneous coordinates of the i-th three-dimensional point; K is the internal reference matrix of the virtual camera; [R|t] is the transformation from the world coordinate system to the camera coordinate system.

[0114] Furthermore, a BIM-assisted indoor positioning system based on cross-domain image retrieval includes:

[0115] Model building module, used to build BIM models;

[0116] A roaming image capture module is used to roam the BIM model using a virtual camera in the Revit software, and to capture roaming images at a predefined frame rate to obtain a BIM rendering image data set;

[0117] A test image acquisition module is used to capture an indoor image of a target building, and use a NAFNet network to remove dynamic blur from the indoor image of the target building to obtain a test image;

[0118] An image synthesis module, used to convert the BIM rendering image dataset into a synthetic image set using the trained improved CycleGAN network;

[0119] A most similar synthetic image retrieval module, used to perform feature extraction, cosine similarity calculation and image retrieval on the test image and the synthetic image set using a pre-trained Swin Transformer model to obtain the most similar synthetic image;

[0120] A mapping module, used to map the most similar synthetic image from two dimensions to three dimensions using a SIFT algorithm to obtain mapping relationship data;

[0121] The position extraction module is used to use the SRPnP algorithm to perform camera six-degree-of-freedom posture estimation on the mapping relationship data to obtain indoor position recognition results.

[0122] Specifically, the detailed steps of a BIM-assisted indoor positioning method based on cross-domain image retrieval are as follows:

[0123] Step 1: Generate a BIM rendering image video stream by roaming with a virtual camera in Revit software, automatically capture it at a predefined frame rate, and construct a BIM rendering image dataset;

[0124] Step 11: Configure the imported BIM model in the software, including wireframe, hatching, consistent colors, and realistic rendering options to ensure high-quality visuals.

[0125] Step 12: Simulate the actual viewing process in the model space by setting the roaming path of the virtual camera, including setting the frame interval and ensuring that the camera can cover all the predetermined walking frames; and control the movement of the pointer to ensure that all frames are traversed, so as to obtain a complete BIM rendering image data set;

[0126] Step 13, save each frame image in XML format and output the BIM rendering image data set;

[0127] Furthermore, users can use Revit’s virtual camera walkthrough feature to extensively navigate within the 3D space of a BIM model; whether by fixed-point rotation or walkthrough, the resulting BIM rendered image video stream is automatically captured at a predefined frame rate; if dealing with larger or more complex environments, the frame rate can be increased to ensure a suitably comprehensive data set.

[0128] Furthermore, in order to preserve the dataset, IO stream operations are used to save the information of each frame in XML format; this facilitates local or database storage for subsequent analysis; and the traversal of all frames is ensured by controlling the movement of the pointer, thus ensuring complete exploration;

[0129] Step 2: Determine whether the indoor image captured in reality needs to be deblurred, and use the nonlinear non-activated network NAFNet to remove the motion blur of the indoor positioning image to improve the clarity and quality of the image;

[0130] Step 21, using the blur discrimination threshold T of the image blur discrimination metric BDM to distinguish between a clear image and a motion blurred image;

[0131] Step 22: Input the image to be deblurred into NAFNet trained with the GoPro public dataset to perform image deblurring.

[0132] Specifically, the image blur discrimination metric BDM and blur discrimination threshold T are calculated as follows:

[0133]

[0134] Step 3: By improving the Cycle Generative Adversarial Network (CycleGAN), the BIM rendering image is converted into a synthetic image with the style of a real interior photo. Figure 3 As shown, the details are as follows:

[0135] Step 31, integrate CBDM into the residual block of the generator, replace the rectified linear unit ReLU activation function with the parameter rectified linear unit PReLU in the convolutional layer of the generator, and use the label smoothing technology in the discriminant module, so as to improve and enhance the CycleGAN network;

[0136] Step 32: Take the BIM rendering images as the training set trainA, and take the indoor shooting images as trainB; input trainA and trainB into the improved CycleGAN model, and after training, transform the texture-free BIM rendering images into realistic indoor shooting style images with obvious textures;

[0137] Step 33: Generator G AB The generator G is responsible for converting the BIM model rendering image into a synthetic image in the style of an indoor shooting image. BA The opposite process is performed, and the discriminator D B It is used to distinguish the synthetic images of the indoor shooting image style generated by the generator from the indoor shooting image; the cycle consistency process is that the generator takes the BIM model rendering image as input, generates the synthetic image of the indoor shooting image style, and inputs the generator G BA Generate the reconstructed image G BA (G AB (a)); The generator in CycleGAN uses adversarial loss For training, the calculation formula is as follows:

[0138]

[0139] Step 34. A Frechette Inception Distance (FID) verification module is integrated into the improved CycleGAN initial training framework; this module is used as a metric to evaluate the similarity between the indoor photos and the generated indoor style synthetic images by carefully examining the statistical properties of computer vision features; this verification is performed intermittently throughout the iteration process; the ideal FID score is 0, indicating that the two sets of images are completely consistent; the lower the score, the higher the similarity; the calculation formula of FID is as follows:

[0140] FID=||μr -μ g || 2 +Tr(∑r+∑g-2(∑r∑g) 1 / 2 )

[0141] Step 4: Use the Swin Transformer model to extract image features from synthetic images and real photos and calculate cosine similarity to achieve image retrieval;

[0142] Step 41: Use the pre-trained Swin Transformer network model to extract the feature vectors (denoted as y1, y2, ..., y k ), store all feature vectors in the offline feature database; where y k represents the k feature vectors extracted from the synthetic image generated by the improved CycleGAN;

[0143] Step 42: randomly select a number of indoor images as testA, use the feature vector of the testA image to search in the offline feature database, and use the pre-trained Swin Transformer network model to extract the feature vector of the indoor image (denoted as x1, x2, ..., x k ); x k represents k feature vectors extracted from the indoor images;

[0144] Step 43, calculate the cosine similarity value between the feature vector of the indoor image and the feature vector of each synthetic image in the offline feature database; sort the calculated similarity values, and take the generated image corresponding to the highest similarity value as the retrieval result; the expression for calculating the cosine similarity is as follows:

[0145]

[0146] Among them, x r represents the number of correctly retrieved images, x represents the total number of query images, and m is the number of feature vectors;

[0147] Step 5: Retrieve the corresponding 3D BIM space coordinates from any 2D pixel coordinates in the synthetic image through the scale-invariant feature transformation (SIFT) algorithm to construct a 2D-3D correspondence relationship; use the SRPnP algorithm to estimate the camera's six-degree-of-freedom posture and realize indoor positioning solution, such as Figure 4 As shown, the details are as follows:

[0148] Step 51: Use the SIFT algorithm to detect and identify scale-invariant local feature points in the synthetic image data set;

[0149] Step 52, using the 3D space geometry as prior knowledge, obtain the corresponding 3D world coordinates from the 2D pixel coordinates through the selection method of UIDoc.Selection.PickObject in Revit;

[0150] Step 53, a database is established to store the two-dimensional pixel coordinates and their three-dimensional space coordinates in correspondence, and this information is persistently stored in XML format;

[0151] Step 54: Utilize the 2D-3D correspondence of SIFT feature points and the internal reference of the camera, use the SRPnP algorithm to calculate the R rotation matrix (3×3) and T translation vector (3×1) from the world frame to the camera frame, obtain the six-degree-of-freedom position and attitude of the camera, and realize indoor positioning solution. The expression of the position and attitude calculation is as follows:

[0152] s i ·p i =K[R|t]P i

[0153] Step 55, calculate the camera positioning error by minimizing the reprojection error; evaluate the camera pose estimation performance based on the positioning error and orientation error; the positioning error refers to the Euclidean distance (m) between the estimated camera position and the ground truth camera position, and the orientation error refers to the angle (°) between the estimated camera observation direction and the ground truth observation direction.

[0154] Preferably, the SRPnP camera pose estimation algorithm used in this embodiment has the smallest average positioning error (1.16 meters) and the smallest average direction error (6.30°) compared with the RPnP, RPnP, EPnP, UPnP, 2DTriPnP, DLT and DLS methods, and the average running time is only 0.36 seconds, ensuring fast real-time response. In addition, SRPnP has advantages in handling various types of data uncertainty and noise, which is particularly important in complex indoor environments.

[0155] Further, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any function designed in this embodiment is implemented. In the implementation process, any function can be completed by an integrated logic circuit of hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with this embodiment can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware.

[0156] refer to Figure 4 , a hardware structure diagram, the electronic device includes: a memory and a processor, the memory stores a computer program, and the processor is communicatively connected to the memory. When the computer program is called, a BIM-assisted indoor camera pose estimation method based on cross-domain image retrieval in the above-mentioned scheme is executed.

[0157] In some possible implementations, the memory may include various media that can store program codes, such as ROM, RAM, magnetic disk, U disk, memory card or optical disk. Specifically, the memory may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory may include at least one program product, which has a group (e.g., at least one) of program modules, which are configured to perform the functions of various embodiments of the present invention. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0158] The beneficial effects of the present invention are as follows:

[0159] The present invention solves the motion blur problem in the image retrieval and positioning process through the NAFNet network; realizes high-similarity synthesis of virtual captured images to real images through the improved CycleGAN network, improves the authenticity and credibility of the synthesized images, and reduces the cross-domain differences between BIM rendered images and real indoor photos; realizes feature extraction and efficient retrieval of images through the Swin Transformer; and improves the accuracy and speed of indoor positioning through the SRPnP algorithm.

[0160] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0161] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A BIM-assisted indoor positioning method based on cross-domain image retrieval, characterized in that: include: Build BIM models; Using a virtual camera in the Revit software to roam the BIM model, and capturing roaming images at a predefined frame rate to obtain a BIM rendering image dataset; Taking an indoor image of a target building, and removing dynamic blur from the indoor image of the target building using a NAFNet network to obtain a test image; Using the trained improved CycleGAN network to convert the BIM rendering image dataset into a synthetic image set; Using a pre-trained Swin Transformer model, performing feature extraction, cosine similarity calculation, and image retrieval on the test image and the synthetic image set to obtain the most similar synthetic image; Using the SIFT algorithm to map the most similar synthetic image from two dimensions to three dimensions, to obtain mapping relationship data; The SRPnP algorithm is used to estimate the six-degree-of-freedom posture of the camera on the mapping relationship data to obtain an indoor position recognition result.

2. A BIM-assisted indoor positioning method based on cross-domain image retrieval according to claim 1, characterized in that: The BIM model is roamed using a virtual camera in the Revit software, and roaming images are captured at a predefined frame rate to obtain a BIM rendering image dataset, including: Configure the generated property options of the Revit software; the property options include: wireframe, hatching, color options, and rendering options; Setting a roaming path of the virtual camera; the roaming path includes: a frame interval and a predetermined walking frame; Based on the Revit software configured with the generation property option, the virtual camera is used to collect images of the BIM model according to the roaming path to obtain the BIM rendering image data set in XML format.

3. The BIM-assisted indoor positioning method based on cross-domain image retrieval according to claim 1 is characterized in that: Take an indoor image of a target building, and use the NAFNet network to remove dynamic blur from the indoor image of the target building to obtain a test image, including: Using a camera to photograph the interior of a target building to obtain an indoor image of the target building; The image fuzzy discrimination metric and fuzzy discrimination threshold of the target building indoor image are calculated; the calculation formula of the image fuzzy discrimination metric is: The calculation formula of the fuzzy discrimination threshold is: Wherein, BDM is the image blur discrimination metric; T is the blur discrimination threshold; H is the image height; W is the image width; g x (i, j) is the gradient of the image in the x direction; g y (i, j) is the gradient of the image in the y direction; N is the total number of gradient values ​​calculated for the image in the x and y directions; G n is the sum of non-zero gradient values; n is the number of scenes; and are respectively the maximum and minimum values ​​of the image blur discrimination metric; Performing blur judgment on the indoor image of the target building according to the image blur discrimination metric and the blur discrimination threshold to obtain blur degree data; When the blur degree data exceeds a preset blur allowable range, the NAFNet network is used to perform dynamic blur removal on the indoor image of the target building to obtain the test image.

4. The BIM-assisted indoor positioning method based on cross-domain image retrieval according to claim 1 is characterized in that: The training process of the improved CycleGAN network includes: The CBDM is integrated into the residual block of the generator of the basic CycleGAN network, the ReLU activation function in the convolutional layer of the generator of the basic CycleGAN network is replaced by the PReLU activation function, and the label smoothing technique is used to optimize the discriminant module of the basic CycleGAN network to obtain an improved CycleGAN network; The pre-collected BIM rendering images are used as the training set, and the pre-collected indoor shooting images are used as the validation set; Using the generator in the improved CycleGAN network to convert the training set into an indoor shooting style, to obtain a generated image set; Using the discriminator in the improved CycleGAN network to judge the authenticity of the generated image set, and obtain a judgment result; When the judgment result is generation failure, the loss value of the generated image set is calculated to obtain an intermediate adversarial loss, and the parameters of the improved CycleGAN network are optimized and iterated using the intermediate adversarial loss to obtain the improved CycleGAN network after iteration. The calculation formula of the adversarial loss is: in, is the adversarial loss; A and B are both source domains; G AB and G BA are all the generators mentioned above; D B is the discriminator; G BA (G AB (a)) is an image in the generated image set; is the real adversarial loss; To generate adversarial loss; The generation similarity of the iteratively completed improved CycleGAN network is calculated to obtain the Frechette starting distance; the calculation formula of the Frechette starting distance is: FID=||μ r -μ g || 2 +Tr(∑r+∑g-2(∑r∑g) 1 / 2 ); Wherein, FID is the Frechette starting distance; μ r , μ g are the feature means of the validation set and the generated image set respectively; ∑r and ∑g are the covariance matrices of the validation set and the generated image set respectively; Tr is the trace; When the Frechette starting distance meets the preset similarity standard, the trained improved CycleGAN network is output.

5. The BIM-assisted indoor positioning method based on cross-domain image retrieval according to claim 1 is characterized in that: The pre-trained SwinTransformer model is used to perform feature extraction, cosine similarity calculation and image retrieval on the test image and the synthetic image set to obtain the most similar synthetic image, including: Using the SwinTransformer model, respectively extracting features from the test image and the synthetic image set to obtain a test feature and a synthetic feature set; Calculating the cosine similarity between the test feature and each synthetic feature in the synthetic feature set to obtain a similarity data set; According to the similarity data set, a synthetic image with the largest similarity value in the synthetic image set is determined as the most similar synthetic image.

6. The BIM-assisted indoor positioning method based on cross-domain image retrieval according to claim 1 is characterized in that: The most similar synthetic image is mapped from two dimensions to three dimensions using the SIFT algorithm to obtain mapping relationship data, including: Using SIFT algorithm to obtain local feature points of the most similar composite image; The three-dimensional world coordinates of the local feature points are obtained by using the UIDoc.Selection.PickObject function in the Revit software; According to the most similar synthetic image, pixel mapping is performed using the three-dimensional world coordinates to obtain the mapping relationship data.

7. The BIM-assisted indoor positioning method based on cross-domain image retrieval according to claim 1 is characterized in that: The calculation formula of the indoor position recognition result is: i ·p i =K[R|t]P i ; Where R is the rotation matrix; t is the translation vector; s i is the unknown proportional factor; p i is the i-th point on the most similar composite image; P i are the homogeneous coordinates of the i-th three-dimensional point; K is the internal reference matrix of the virtual camera; [R|t] is the transformation from the world coordinate system to the camera coordinate system.

8. A BIM-assisted indoor positioning system based on cross-domain image retrieval, characterized in that: include: Model building module, used to build BIM models; A roaming image capture module is used to roam the BIM model using a virtual camera in the Revit software, and to capture roaming images at a predefined frame rate to obtain a BIM rendering image data set; A test image acquisition module is used to capture an indoor image of a target building, and use a NAFNet network to remove dynamic blur from the indoor image of the target building to obtain a test image; An image synthesis module, used to convert the BIM rendering image dataset into a synthetic image set using the trained improved CycleGAN network; A most similar synthetic image retrieval module, used to perform feature extraction, cosine similarity calculation and image retrieval on the test image and the synthetic image set using a pre-trained Swin Transformer model to obtain the most similar synthetic image; A mapping module, used to map the most similar synthetic image from two dimensions to three dimensions using a SIFT algorithm to obtain mapping relationship data; The position extraction module is used to use the SRPnP algorithm to perform camera six-degree-of-freedom posture estimation on the mapping relationship data to obtain indoor position recognition results.

Images