Indoor positioning system and method

Abstract

The invention relates to the technical field of indoor positioning, and discloses an indoor positioning system and method. The system comprises an acquisition unit used for converting a target video acquired in real time into video frames according to a first time interval; the feature extraction unit is used for extracting an initial feature from a current video frame through a pre-trained convolutional neural network, and processing the initial feature to obtain a target feature; the calculation module is used for constructing an index of a K-d tree structure according to the image features of the sample pictures; the matching unit is used for controlling the calculation module to execute a K nearest neighbor algorithm and matching the image feature most similar to the target feature in the index; and the positioning unit is used for acquiring corresponding position information based on the most similar image features and positioning the current video frame. The system can well adapt to an indoor complex environment, and high-precision real-time indoor positioning is realized.

Description

Technical Field

[0001] The present application relates to the field of indoor positioning technology, and in particular to an indoor positioning system and method. Background Art

[0002] Indoor positioning technology is widely used in smart security, logistics and warehousing, industrial manufacturing, and assisted navigation. Compared with traditional outdoor positioning systems such as GPS (Global Positioning System), accurate positioning in indoor environments is more difficult due to issues such as occlusion, multipath interference, and signal attenuation.

[0003] Traditional indoor positioning technologies typically rely on wireless signal measurements, such as triangulation and fingerprint matching based on Wi-Fi, Bluetooth, infrared, or ultra-wideband signals. These methods are limited by signal interference, occlusion, and reflection, making it difficult to guarantee positioning accuracy in dynamic indoor environments. Alternatively, there are indoor positioning technologies based on inertial navigation, which use accelerometers and gyroscopes to collect device motion information to infer position. However, accumulated sensor errors can lead to drift. Therefore, achieving high-precision, real-time positioning in indoor environments is a pressing issue. Summary of the Invention

[0004] In view of this, the present application aims to propose an indoor positioning system and method to achieve high-precision real-time positioning in an indoor environment.

[0005] To achieve the above objectives, the technical solutions of this application are as follows:

[0006] A first aspect of an embodiment of the present application provides an indoor positioning system, the system comprising:

[0007] An acquisition unit, configured to convert the target video acquired in real time into video frames according to a first time interval;

[0008] A feature extraction unit, comprising a feature extractor and a recognizer; the feature extractor is used to extract initial features from the current video frame using a pre-trained convolutional neural network; the initial features are convolutional feature maps or feature vectors; the recognizer is used to process the initial features to obtain target features;

[0009] A calculation module is configured to construct a Kd tree structure index based on the image features of the sample images; each node in the index stores the image features of a sample image; the closer the nodes in the index, the higher the similarity; and the sample images contain at least one of the following detection objects: indoor buildings, indoor objects, and indoor signs;

[0010] The matching unit includes a classification and recognition module and a storage submodule; the storage submodule is used to store sample images and corresponding image features; the classification and recognition module is used to control the calculation module to execute the K-nearest neighbor algorithm to match the image feature most similar to the target feature in the index;

[0011] The positioning unit includes a positioning submodule and a position information library; the position information library is used to store feature data and corresponding position information; the positioning submodule is used to obtain corresponding position information from the position information library based on the most similar image feature, and locate the current video frame based on the position information.

[0012] Optionally, the feature extractor is specifically configured to process the video frame through a convolutional neural network to generate a convolutional feature map;

[0013] The matching unit further includes an extraction submodule for extracting feature data of some channels from the convolution feature map using a random sampling algorithm to generate a simplified feature map;

[0014] The feature extractor is further configured to aggregate the simplified feature map into a feature vector;

[0015] The identifier is specifically used to process the convolution feature map or the feature vector as an initial feature to obtain the target feature.

[0016] Optionally, the classification and recognition module is configured to control the calculation module to execute a K-nearest neighbor algorithm to match an image feature in the index that is most similar to the target feature, specifically including:

[0017] Determine the image to be matched based on all sample images;

[0018] Controlling the calculation module to traverse the to-be-matched nodes where the image features of each to-be-matched picture are located based on the index, and calculating the Euclidean distance between the image features of each to-be-matched node and the target feature;

[0019] The image feature corresponding to the smallest Euclidean distance that is not greater than the distance threshold is screened out and determined as the most similar image feature.

[0020] Optionally, the identifier is further configured to analyze the initial features to obtain a corresponding target type;

[0021] The storage submodule is further used to store the type of each sample image;

[0022] The classification and recognition module is configured to determine the to-be-matched pictures based on all the sample pictures, specifically including: based on the target type, taking the sample pictures of the same type as the to-be-matched pictures;

[0023] Each node in the index stores the picture ID corresponding to the image feature;

[0024] The calculation module is used to traverse the to-be-matched nodes where the image features of each to-be-matched picture are located based on the index, and calculate the Euclidean distance between the image features of each to-be-matched node and the target feature, specifically including:

[0025] Based on the picture IDs corresponding to all the pictures to be matched, search the index to determine the location of the corresponding node to be matched;

[0026] Based on the position of each to-be-matched node, the Euclidean distance between the image feature of each to-be-matched node and the target feature is calculated by traversing the layers from top to bottom.

[0027] Optionally, the classification and recognition module is configured to screen out image features corresponding to the smallest Euclidean distance that is not greater than a distance threshold and determine them as the most similar image features, specifically including:

[0028] Compare the Euclidean distance of each node to be matched with the distance threshold. If the Euclidean distance of the current node is less than or equal to the distance threshold, add the image ID stored in the current node to the result set. If the Euclidean distance of the current node is greater than the Euclidean distance in the result set, skip traversing the child nodes of the node.

[0029] Based on the node corresponding to the picture ID in the result set, further traverse the child nodes in the next layer of the node; if the Euclidean distance of the child node is less than or equal to the Euclidean distance corresponding to the picture ID in the result set, then add the picture ID corresponding to the child node to the result set and overwrite the old picture ID;

[0030] After traversing all nodes to be matched, if the result set is empty, it is determined that there is no image feature similar to the target feature; if the result set is not empty, the image feature corresponding to the picture ID currently stored in the result set is determined as the most similar image feature.

[0031] Optionally, the location information library is further used to cache corresponding location information after locating the current video frame;

[0032] The system further includes a display module for displaying position information corresponding to the current video frame on a user interface;

[0033] The positioning submodule is further configured to obtain all cached location information within a third time interval from the current time, and mark the real-time motion trajectory on the map based on the order of all the location information; the third time interval is at least twice as long as the first time interval;

[0034] The display module is also used to display the map in real time on the user interface.

[0035] Optionally, the positioning submodule is further configured to obtain, when there is no sample picture similar to the video frame, all position information cached within a third time interval from the current time, and obtain a motion trend based on analysis of all position information;

[0036] The position information corresponding to the current video frame is predicted based on the motion trend and the position information corresponding to the previous video frame.

[0037] Optionally, the indoor positioning system further includes a pre-processing module for performing visualization processing on the video frames to generate data visual images;

[0038] The identifier is further configured to use the data visual image as auxiliary information, process the initial features and the data visual image, and obtain the target features.

[0039] Optionally, the matching unit further includes an index management module for storing the index of the Kd tree structure and creating mapping information for each sample image; the mapping information includes: the image ID, file name and storage path of the sample image;

[0040] The index management module is also used to scan the storage path of each sample image at a second time interval to detect whether a new file name appears; if a new file name appears, the corresponding old mapping information is searched based on the storage path corresponding to the new file name; the image ID in the old mapping information is obtained, and new mapping information is created in combination with the new file name and the storage path, and the old mapping information is overwritten.

[0041] According to a second aspect of an embodiment of the present application, there is provided an indoor positioning method, which is applied to the system provided in the first aspect of the embodiment of the present application, and the method includes:

[0042] Converting the target video captured in real time into video frames according to a first time interval;

[0043] Extracting initial features from the current video frame using a pre-trained convolutional neural network; the initial features are convolutional feature maps or feature vectors;

[0044] Processing the initial features to obtain target features;

[0045] Constructing a Kd tree structure index based on the image features of the sample images; each node in the index stores the image features of a sample image; the closer the nodes in the index are, the higher the similarity; the sample images contain at least one of the following detection objects: indoor buildings, indoor objects, and indoor signs;

[0046] Using a k-nearest neighbor algorithm, matching the image feature that is most similar to the target feature in the index;

[0047] Corresponding position information is acquired based on the most similar image feature, and the current video frame is positioned based on the position information.

[0048] Optionally, before extracting initial features from the current video frame through a pre-trained convolutional neural network, the following steps are further included:

[0049] Acquire multiple images containing detection objects as sample images; the detection objects include at least: indoor buildings, indoor objects, and indoor signs;

[0050] Add corresponding image ID and type label to each sample image, and store all sample images;

[0051] Based on the sample images, a convolutional neural network is trained using a transfer learning method, and the fully connected layer is removed from the trained convolutional neural network, and a feature extractor is constructed based on the remaining network;

[0052] The image features of each sample picture are extracted by the feature extractor and stored corresponding to the sample picture.

[0053] The indoor positioning system provided by the present application utilizes the feature extraction capability and efficient classification characteristics of convolutional neural networks, combined with the K-nearest neighbor algorithm to achieve fast and high-precision indoor positioning. First, the acquisition unit converts the real-time video into video frames according to a first time interval, and then the feature extraction unit processes the video frames, extracts the initial features from the video frames and analyzes and processes them to obtain target features. The matching unit uses the target features to compare with the image features of pre-stored sample images, so as to obtain the image features that are most similar to the target features. In the present application, the storage submodule pre-stores sample images and corresponding image features of detection objects including indoor buildings, indoor objects, indoor signs, etc., and by comparing the target features with the image features of the sample images, the position of the current video frame can be determined based on the position information corresponding to the most similar image features. Compared with traditional positioning solutions that rely on wireless signals or sensors, this solution adapts well to complex indoor environments and achieves high-precision real-time indoor positioning.

[0054] In this application, the comparison of target features adopts the K nearest neighbor algorithm, constructs an index of the Kd tree structure based on the calculation module, and performs feature comparison based on the arrangement of the nodes in the index. Since the closer the distance between the nodes in the index, the higher the similarity between the corresponding image features, when performing feature matching, the sub-node branches for further matching can be determined based on the similarity of each node in each layer of branches. For nodes with low similarity, the traversal of their sub-nodes can be directly skipped. Therefore, compared with the traditional solution, this application does not need to traverse the image features of all sample images, saving a lot of computing resources, achieving efficient matching, and further improving the real-time performance of indoor positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0056] Figure 1 is a schematic diagram of an indoor positioning system proposed in one embodiment of the present application;

[0057] Figure 2 This is a flowchart of constructing a feature extractor in one embodiment of the present application;

[0058] Figure 3 This is a workflow diagram of an indoor positioning system in one embodiment of the present application;

[0059] Figure 4 This is a flow chart of an indoor positioning method proposed in one embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0061] It should be understood that references throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" throughout this specification do not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0062] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0063] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with certain aspects as detailed herein.

[0064] It should be noted that, unless there is any conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0065] Traditional indoor positioning methods based on wireless signal measurement are limited by signal interference and complex indoor environments, making it difficult to ensure positioning accuracy. Feature point matching or template matching methods have a huge computational load and low efficiency, and are not sensitive and robust enough to image clarity and environmental changes, making it impossible to achieve real-time positioning and high precision. This solution is based on deep learning visual recognition technology, uses convolutional neural networks to perform feature comparison of visual images, and combines the K-nearest neighbor algorithm to construct a Kd tree structure index to achieve fast retrieval and similarity matching, which can achieve high-precision real-time positioning while taking into account both high precision and real-time positioning. The following will refer to the accompanying drawings and combine with the embodiments to explain this application in detail.

[0066] Figure 1 Schematic diagram of an indoor positioning system proposed in one embodiment of the present application. Figure 1 As shown, the system includes:

[0067] An acquisition unit, configured to convert the target video acquired in real time into video frames according to a first time interval;

[0068] A feature extraction unit, comprising a feature extractor and a recognizer; the feature extractor is used to extract initial features from the current video frame using a pre-trained convolutional neural network; the initial features are convolutional feature maps or feature vectors; the recognizer is used to process the initial features to obtain target features;

[0069] A calculation module is configured to construct a Kd tree structure index based on the image features of the sample images; each node in the index stores the image features of a sample image; the closer the nodes in the index, the higher the similarity; and the sample images contain at least one of the following detection objects: indoor buildings, indoor objects, and indoor signs;

[0070] The matching unit includes a classification and recognition module and a storage submodule; the storage submodule is used to store sample images and corresponding image features; the classification and recognition module is used to control the calculation module to execute the K-nearest neighbor algorithm to match the image feature most similar to the target feature in the index;

[0071] The positioning unit includes a positioning submodule and a position information library; the position information library is used to store feature data and corresponding position information; the positioning submodule is used to obtain corresponding position information from the position information library based on the most similar image feature, and locate the current video frame based on the position information.

[0072] In this embodiment, a large number of sample images containing the detection object are collected in advance, each sample image is assigned a unique image ID, and the image features of the sample images are extracted and the corresponding location information is added. The sample images (with unique image IDs) and the corresponding image features are pre-stored in the storage submodule, and the location information corresponding to the image features is stored in the location information library.

[0073] The positioning system employs a flexible modular design, with each module independently developed and debugged, facilitating subsequent functional upgrades, algorithm optimization, and integration of new features. The acquisition unit captures the target video stream in real time and converts the video into video frames at first time intervals. Depending on the application requirements, the video frames can be image files in a variety of formats, which are not limited in this embodiment.

[0074] The feature extraction unit is a deep learning convolutional neural network, including a feature extractor and a recognizer. The feature extractor processes the video frame image using a pre-trained convolutional neural network to extract initial features. The initial features can be two-dimensional convolutional feature maps or one-dimensional feature vectors. The recognizer analyzes the initial features and further extracts deep one-dimensional feature vectors (i.e., target features) for subsequent feature matching. This solution utilizes the feature extraction capabilities and efficient classification characteristics of deep learning convolutional neural networks to perform multi-layer analysis and processing on the feature extraction of video frames. This can reduce the impact of environmental changes on positioning accuracy and improve the system's adaptability in complex and changing indoor environments. For example, in the case of changing light or blurred images, the positioning mechanism based on image features can provide stable recognition results. Optionally, the feature extraction module can also use a more efficient deep learning model. For example, deep learning networks such as ResNet residual networks, MobileNet networks, and EfficientNet networks can also be used as feature extraction modules.

[0075] In this embodiment, sample images and corresponding image features are pre-stored in the storage submodule. Sample images are pre-collected images containing detection objects, including various types of detection objects (indoor buildings, indoor objects, and indoor signs, etc.), such as elevators, floor signs, and water dispensers.

[0076] To facilitate image feature matching, the calculation module constructs an index of a Kd tree (a binary tree) structure based on the sample images and image features pre-stored in the storage submodule. Each node in the index stores feature data corresponding to a sample image. Moreover, the similarity between the image features stored in nodes that are closer in the index is higher. Based on the index of the Kd tree structure, the classification and recognition module controls the calculation module to execute the K nearest neighbor algorithm, matching the target feature with the image features stored in the index, and determining the image feature that is most similar to the target feature, that is, determining the sample image that is most similar to the current video frame. After determining the most similar image feature, the positioning submodule obtains the corresponding position information from the position information library based on the image feature, and determines the position information as the position information of the current video frame.

[0077] In this embodiment, a pre-trained convolutional neural network is used to efficiently extract and classify the image features of video frames. Combined with the K-nearest neighbor algorithm and the Kd tree structure index, rapid retrieval and feature matching of target features are achieved. Compared to traditional positioning solutions that rely on wireless signals or sensors, this solution adapts well to complex indoor environments and achieves high-precision real-time indoor positioning. Furthermore, since the closer the distance between nodes in the Kd tree structure index, the higher the similarity between the corresponding stored image features, when performing feature matching based on the nodes in the index, the traversed node branches can be quickly screened based on the similarity of each layer of branch nodes, and node branches with low similarity can be directly skipped without calculating the similarity. Therefore, compared to the traditional solution of traversing all sample images in sequence, this solution can save a lot of computing resources and further improve the real-time performance of indoor positioning.

[0078] This embodiment constructs a feature extractor based on the VGG16 convolutional neural network model. The VGG16 model has a deep convolutional structure and its parameters are transferable, which can effectively reduce the demand for large-scale training data. The structure of the VGG16 model includes 13 convolutional layers, 5 pooling layers and 3 fully connected layers, and its convolution kernel size is fixed to 3×3, with a step size of 1. The pooling layer adopts the maximum pooling method, the pooling window is 2×2, and the step size is 2. The convolution layer of the VGG16 model extracts spatial local features through the local receptive field, and generates feature representations with semantic information after deepening layer by layer. The fully connected layer further aggregates features for final classification. In this embodiment, the VGG16 model is fine-tuned by training the model based on the collected sample images in advance through the transfer learning method to construct a feature extraction unit. Transfer learning effectively reduces the dependence on a large amount of labeled data by utilizing pre-trained weights while speeding up the training process, improving the generalization ability and adaptability of the model, and ensuring the stability of the system in diverse environments.

[0079] Figure 2 This is a flow chart of constructing a feature extractor in one embodiment of the present application. Figure 2 As shown, in this embodiment, the construction process of the feature extractor is as follows:

[0080] (1) First, prepare and annotate the dataset. In this embodiment, a sample dataset is constructed based on the collected sample images. A corresponding type label is added to each sample image, and the images are stored in the storage submodule according to their type. The sample dataset is divided into a training set and a test set. The training set is used to guide the model to learn feature representations, and the test set is used to evaluate and test the classification effect.

[0081] (2) Then, the convolutional neural network to be trained (i.e., the VGG16 model) is initialized, and the training set is loaded on the pre-trained VGG16 model for fine-tuning using the transfer learning method. The key to transfer learning is to freeze some convolutional layers (usually shallow convolutional layers) and only update high-level parameters to adapt to the new task. The cross-entropy loss function is used for optimization during training, which is defined as follows:

[0082]

[0083] Among them, N is the total number of training samples; C is the total number of categories; y i,c is the indicator function of the true category of the i-th sample, which takes the value of 1 when the sample belongs to category c, otherwise it is 0; The network predicts the class probability. By optimizing the loss function, the network adjusts its parameters to make the predicted probability as close to the true distribution as possible, thereby improving classification accuracy.

[0084] (3) Use the test set to verify the classification results of the model. The trained model analyzes the sample images in the test set, outputs the classification results, and compares them with the types pre-stored in the storage submodule. For example, an indoor scene contains objects A, B, and C, and the corresponding category labels are marked as 0, 1, and 2 respectively. After the model training is completed, a picture containing object A is input. The network extracts the feature map through convolution operation and outputs category 0, and classifies the picture as object A. During the iterative training of the model, the evaluation indicators such as accuracy, recall rate, and F1 value are used to judge whether the training accuracy of the model meets the requirements. If it meets the requirements, the training of the model is stopped.

[0085] (4) Remove the fully connected layers from the trained VGG16 model, retaining only the convolutional layers and pooling layers to obtain a feature extractor. In this embodiment, the feature extractor is used to extract feature information from the image. Specifically, the feature extractor performs a convolution operation on the input video frame to generate a corresponding convolution feature map. The specific expression is as follows:

[0086] f i,j,k =σ(∑ m,n,p w m,n,p,k ·x i+m,j+n,p +b k );

[0087] Among them, f i,j,k is the value of the kth channel at position (i, j) of the output feature map; w m,n,p,k is the convolution kernel parameter; x i+m,j+n,p is the pixel value at the corresponding position of the input feature map; b k is the bias term; σ is the activation function. The above convolution operation implements linear weighting and nonlinear activation of local features, which can extract the deep features of the input image.

[0088] As an embodiment of the present application, the feature extractor is specifically configured to process the video frame through a convolutional neural network to generate a convolutional feature map;

[0089] The matching unit further includes an extraction submodule for extracting feature data of some channels from the convolution feature map using a random sampling algorithm to generate a simplified feature map;

[0090] The feature extractor is further configured to aggregate the simplified feature map into a feature vector;

[0091] The identifier is specifically used to process the convolution feature map or the feature vector as an initial feature to obtain the target feature.

[0092] In this embodiment, the feature extractor processes the video frame and extracts a two-dimensional convolutional feature map from it. The convolutional feature map is used as the initial feature input to the recognizer, or the convolutional feature map is further compressed and aggregated to generate a low-dimensional feature vector with a smaller data volume as the initial feature. Specifically, after the input video frame passes through the feature extractor, the generated convolutional feature map can be represented as a three-dimensional tensor F∈R H×W×C , where H and W are the height and width of the convolutional feature map, respectively, and C is the number of feature channels. Since convolutional neural networks typically have a large number of channels (for example, the middle layer of VGG16 may contain hundreds of channels), directly using the convolutional feature map for subsequent processing results in high computational complexity.

[0093] In one embodiment, the amount of data is reduced in the feature extraction stage to improve processing efficiency. Considering that some feature information contributes less to the classification task, the channel information of the convolution feature map can be simplified, and a random sampling algorithm is used to extract feature data of some channels from all channels, thereby reducing redundant data and generating a simplified feature map. Furthermore, the feature extractor aggregates based on the simplified feature map, extracts more significant feature information, and generates a one-dimensional feature vector, thereby further improving the efficiency of subsequent feature matching while ensuring the accuracy of feature matching. In addition, the use of a random sampling algorithm can also improve the generalization ability of the feature extractor through randomness. The recognizer processes the aggregated feature vector, which reduces the amount of calculation and improves processing efficiency compared to directly processing the convolution feature map.

[0094] In this embodiment, the steps of compressing and aggregating the convolution feature map to obtain the feature vector are as follows:

[0095] (1) Based on the original convolutional feature map, k channels are randomly selected from the total C channels, which are represented as the set S = {s1, s2, ..., s k}, where s i ∈{1,2,…,C};

[0096] The generated simplified feature map is represented as:

[0097] (2) Simplify the feature map F S Normalization and data augmentation are performed. The purpose of normalization is to stabilize the distribution of eigenvalues and improve the convergence speed of the training process. The simplified feature map after normalization is shown as follows:

[0098]

[0099] Among them, F i,j,k To simplify the value of the kth channel of the feature map at position (i, j), μ k and σ kare the mean and standard deviation of the kth channel respectively. The normalized simplified feature map F′ has a distribution with zero mean and unit variance, which helps to eliminate the influence of feature value scale differences;

[0100] (3) Furthermore, the simplified feature map is used as the input of the feature extractor to extract a more compact feature representation. In this embodiment, a dimensionality reduction technique or a specific operator can be used to complete the feature extraction of the simplified feature map. For example, a global pooling operation (maximum pooling or average pooling) is applied to the simplified feature map to aggregate the two-dimensional spatial features into a one-dimensional vector. The specific expression is:

[0101] v k =pool(F′ i，j，k );

[0102] Among them, pool is the pooling operation (such as maximum pooling max or average pooling mean); v k is the feature map F i ' ,j,k The final output one-dimensional feature vector is: v = [v1, v2, ..., v k ].

[0103] For example, a video frame undergoes convolution in the feature extractor, generating a feature map tensor of size 14×14×512. A random sampling algorithm is used to select 10 feature channels, and normalization and max pooling are performed on each channel, ultimately generating a one-dimensional feature vector of size 10×1. By extracting channels and aggregating features from the original convolutional feature map, redundant data is effectively reduced while retaining the most relevant information for the recognizer to classify. This improves the recognizer's computational efficiency while ensuring classification accuracy, thereby enhancing positioning efficiency.

[0104] As an implementation manner of the present application, the identifier is further configured to analyze the initial features to obtain a corresponding target type;

[0105] The storage submodule is further used to store the type of each sample image;

[0106] The classification and recognition module is configured to determine the to-be-matched pictures based on all the sample pictures, specifically including: based on the target type, taking the sample pictures of the same type as the to-be-matched pictures;

[0107] Each node in the index stores the picture ID corresponding to the image feature;

[0108] The calculation module is used to traverse the to-be-matched nodes where the image features of each to-be-matched picture are located based on the index, and calculate the Euclidean distance between the image features of each to-be-matched node and the target feature, specifically including:

[0109] Based on the picture IDs corresponding to all the pictures to be matched, search the index to determine the location of the corresponding node to be matched;

[0110] Based on the position of each to-be-matched node, the Euclidean distance between the image feature of each to-be-matched node and the target feature is calculated by traversing the layers from top to bottom.

[0111] In one embodiment, the recognizer analyzes the initial features and outputs the target features and corresponding target types. The storage submodule then stores the image features of the sample images and their corresponding types. During feature matching, the classification recognition module first screens all sample images based on the target type, selecting sample images of the same type as the images to be matched. Subsequent feature matching is then performed only on the image features of the images to be matched, thereby reducing the computational effort required for feature matching and improving matching efficiency.

[0112] Each node in the index stores an image feature and its corresponding image ID. After selecting the images to be matched, the calculation module searches the index based on their IDs to determine the location of each node. Traversing the nodes from top to bottom, the Euclidean distance between the image features of each node and the target features is calculated to determine the degree of similarity. Furthermore, based on the degree of similarity, the subnode branches of the node to be matched in the next layer are determined.

[0113] As an implementation of the present application, constructing a Kd tree structure index based on image features of sample images includes:

[0114] The image features of all sample images are added to a root set, and subsets are divided based on the root set, specifically including:

[0115] Starting from the first dimension and increasing layer by layer, based on the median of the eigenvalues of the dth dimension, image features with eigenvalues less than the median are divided into the left subtree set, and image features with eigenvalues greater than or equal to the median are divided into the right subtree set; where d∈(1,2,...,m), m is the maximum dimension of the image features;

[0116] When the number of image features in the left subtree set or the right subtree set is greater than 1, further dividing the corresponding subsets based on the median of the feature values of the current dimension in the left subtree set or the median of the feature values of the current dimension in the right subtree set until all subsets contain only one image feature;

[0117] When all subsets contain only one image feature, stop dividing new subsets and build an index of the Kd tree structure based on the hierarchical relationship of the current subsets.

[0118] In this embodiment, the similarity between images is measured by calculating the Euclidean distance. Since all sample images are directly traversed to calculate the Euclidean distance between the image features of the sample images and the target features, the computational overhead increases significantly when the number of images is large. To improve matching efficiency, this embodiment constructs an index structured by a Kd tree to reduce the number of image features that need to be traversed. The Kd tree is a binary tree structure designed for multidimensional data, in which each node stores only the feature vector of a sample image.

[0119] Specifically, the construction process of the Kd tree in this embodiment is as follows:

[0120] (1) Take all the image feature vectors in the image library as input data and recursively divide the data set according to their d-th dimension eigenvalues, where d represents the component index of the feature vector, and loop;

[0121] (2) Each time the data is partitioned, the median of the dth dimension in the data set is selected, and the data less than the median is assigned to the left subtree, and the data greater than or equal to the median is assigned to the right subtree;

[0122] (3) Repeat the above process until each subtree contains only one data point or meets the set depth condition.

[0123] The constructed Kd tree can organize data in a hierarchical manner, significantly reducing the number of sample point traversals. To match target features using the K-nearest neighbor algorithm, we first need to construct an index of the Kd tree structure. Then, based on this index, we recursively construct the index of the Kd tree structure based on the image features (one-dimensional feature vectors) of each sample image. The image features of the sample image are one-dimensional feature vectors, whose dimensions are determined by the number and size of the convolution kernels. The specific steps are as follows:

[0124] (1) Add the image features (one-dimensional feature vectors) of all sample images to the root set, and recursively divide the root set into subsets;

[0125] (2) Starting from d = 1, after each subset is divided, let d = d + 1, d∈(1,2,...,m) represents the component dimension of the feature vector, and m is the maximum dimension of the feature vector. For example, for a feature vector of size 10×1, m = 10. Each time the feature vector is divided, the median of the d-th dimension of the set is selected. The image features with the d-th dimension less than the median are assigned to the subset of the left tree, and the image features with the d-th dimension greater than or equal to the median are assigned to the subset of the right tree;

[0126] (3) Repeat the above process. After the subsets are divided based on the eigenvalues of the mth dimension, if the number of image features remaining in the subset is greater than 1, set d = 1 and restart the cycle to continue dividing the corresponding subsets until only 1 image feature remains in each subset.

[0127] The constructed index organizes image features in a hierarchical node format. Feature matching based on this index can significantly reduce the number of nodes traversed, improving matching efficiency. Optionally, after the sample image is stored in the storage submodule, the computation module can read the image features from the storage submodule and pre-build a Kd-tree-structured index. This eliminates the need to rebuild the index when the system subsequently performs real-time positioning tasks, further improving matching efficiency.

[0128] As an embodiment of the present application, the classification and recognition module is used to control the calculation module to execute the K-nearest neighbor algorithm to match the image feature most similar to the target feature in the index, specifically including:

[0129] Determine the image to be matched based on all sample images;

[0130] Controlling the calculation module to traverse the to-be-matched nodes where the image features of each to-be-matched picture are located based on the index, and calculating the Euclidean distance between the image features of each to-be-matched node and the target feature;

[0131] The image feature corresponding to the smallest Euclidean distance that is not greater than the distance threshold is screened out and determined as the most similar image feature.

[0132] In this embodiment, the similarity between images is determined by calculating the Euclidean distance between the image features of the sample image and the target features. Specifically, assuming that the dimension of the feature vector output by the recognizer is m, it means that the feature information of each image is compressed into an m-dimensional vector. The Euclidean distance is used as the metric to calculate the distance between it and the sample features, as follows:

[0133] Where x=(x1,x2,…,x m )、y=(y1,y2,…,y m ) are the target features of the video frame and the feature vectors of the sample image, m is the maximum dimension of the feature vector, x i and y i are the components of the i-th dimension in the feature vector. The smaller the Euclidean distance, the more similar the two images are.

[0134] In this embodiment, the classification and recognition module controls the calculation module to execute the K nearest neighbor algorithm, traverses all nodes where the images to be matched are located based on the constructed index, and calculates the Euclidean distance between the feature vector of each image to be matched and the target feature in a layer-by-layer traversal from top to bottom (i.e., from the parent node to the child node). The degree of similarity between the image to be matched and the video frame is measured by a distance threshold. When the Euclidean distance is less than or equal to the distance threshold, it is determined that the sample image corresponding to the feature vector stored in the current node is similar to the video frame. Further, the feature vectors of the image to be matched in the next layer of child nodes of the current node are traversed, and finally a feature vector that meets the distance threshold condition and has the smallest Euclidean distance is found, that is, the sample image most similar to the video frame is found.

[0135] As an embodiment of the present application, the classification and recognition module is used to screen out image features corresponding to the smallest Euclidean distance not greater than a distance threshold and determine them as the most similar image features, specifically including:

[0136] Compare the Euclidean distance of each node to be matched with the distance threshold. If the Euclidean distance of the current node is less than or equal to the distance threshold, add the image ID stored in the current node to the result set. If the Euclidean distance of the current node is greater than the Euclidean distance in the result set, skip traversing the child nodes of the node.

[0137] Based on the node corresponding to the picture ID in the result set, further traverse the child nodes in the next layer of the node; if the Euclidean distance of the child node is less than or equal to the Euclidean distance corresponding to the picture ID in the result set, then add the picture ID corresponding to the child node to the result set and overwrite the old picture ID;

[0138] After traversing all nodes to be matched, if the result set is empty, it is determined that there is no image feature similar to the target feature; if the result set is not empty, the image feature corresponding to the picture ID currently stored in the result set is determined as the most similar image feature.

[0139] In this embodiment, a Kd tree structure index is constructed based on sample images to achieve efficient feature retrieval and significantly reduce computational overhead. The classification and recognition module controls the computation module to execute the K nearest neighbor algorithm, performing node traversal and feature matching based on the Kd tree structure index to determine the image feature that is most similar to the target feature. The specific process is as follows:

[0140] (1) Determine the root node to be matched based on the location of the node to be matched. In this embodiment, when the recognizer outputs the type of the current video frame, the classification recognition module pre-screens nodes corresponding to images of the same type from all sample images based on the type as nodes to be matched. If the recognizer does not output a type, the classification recognition module selects nodes corresponding to all sample images as nodes to be matched;

[0141] (2) Starting from the root node to be matched, traverse the child nodes layer by layer, calculate the Euclidean distance between the image feature stored in the node and the target feature, and compare the calculated result with the distance threshold. This step is to determine the distance between the target feature and the segmentation plane of the current node. If the calculated result is less than or equal to the distance threshold, it means that the two are similar, then the image feature is used as the current most similar image feature, and the corresponding picture ID is added to the result set. If the calculated result is greater than the distance threshold, it means that the two are not similar, then stop traversing the node branch, and subsequently skip processing the image features stored in the child nodes of the node, thereby saving computational effort and speeding up the matching process; if the new Euclidean distance calculated during the traversal process is less than the Euclidean distance corresponding to the image feature in the result set, then the image ID corresponding to the new Euclidean distance is added to the result set and replaces the old image ID in the result set. In this way, it is ensured that the result set always contains the picture ID corresponding to the most similar image feature currently matched;

[0142] (3) Repeat the above process until the current node does not have a next-level child node, that is, all nodes to be matched are traversed. When the result set is not empty, the image feature corresponding to the image ID in the current result set is determined as the most similar image feature. In addition, there may be occlusion or dim light in the video frame, making the image difficult to distinguish. The feature extraction unit cannot output accurate target features, and thus may not be able to match similar sample features. That is, in the feature matching process, the Euclidean distance between the image features of the traversed image to be matched and the target features is greater than the distance threshold. After the traversal is completed, the result set is empty, and it is determined that there are no similar image features.

[0143] As an implementation manner of the present application, the location information library is further used to cache corresponding location information after locating the current video frame;

[0144] The system further includes a display module for displaying position information corresponding to the current video frame on a user interface;

[0145] The positioning submodule is further configured to obtain all cached location information within a third time interval from the current time, and mark the real-time motion trajectory on the map based on the order of all the location information; the third time interval is at least twice as long as the first time interval;

[0146] The display module is also used to display the map in real time on the user interface.

[0147] In one embodiment, the positioning system further includes a display module for displaying the location information output by the positioning submodule on a user interface. In this embodiment, the display module can be a mobile client (such as a mobile phone or other communication device) or a fixed display device (such as a display screen, etc.). In this embodiment, the indoor positioning system can return the real-time positioning location information to the user through the client or the server; the client includes client programs such as mobile phones, tablet computers, and web browsers, and the server includes programs such as Windows or Mac server programs. After the user uses the client to obtain the user's own location information, the user can use the mouse (or through touch screen gestures) on the client to locate the location information. Specifically, the user can accurately locate the target of interest through the mouse or touch screen gestures, and can also obtain the user's current location information in real time. Optionally, the positioning submodule also directly obtains the current video frame image from the acquisition unit, and when outputting the current location information, it simultaneously outputs the video frame image to the display module.

[0148] For example, in a large indoor shopping mall, multi-story office building, or large exhibition venue, users can use their mobile devices (such as mobile phones and tablets) to deploy the system's application. Users can use "mouse or touchscreen gestures" to focus on the target of interest (such as a specific store, conference room, exhibit, etc.). The system quickly displays the target's location in a graphical and intuitive manner, making it easy for users to find the target quickly.

[0149] For example, when a user is looking for a store in a shopping mall, he or she needs to know his or her real-time location. By focusing on his or her real-time location through "mouse or touch screen gestures", the system will locate the user's current location in real time to provide clear location guidance and help the user better plan the route to the store.

[0150] In addition, the location information library is used to cache the location information output by the positioning submodule within a recent period (the third time interval). Furthermore, the positioning submodule can read the sequence of locations cached in the location information library for the recent period and mark the corresponding motion trajectory on the map, displaying it in real time through the display module, allowing the user to clearly and intuitively understand the movement of the located object. In order to have sufficient location information to analyze the motion trajectory, the location information library must cache more than 2 locations, that is, the third time interval must be greater than twice the first time interval.

[0151] As an embodiment of the present application, the positioning submodule is further configured to obtain, in the absence of a sample image similar to the video frame, all cached position information within a third time interval from the current time, and obtain a motion trend based on analysis of all the position information;

[0152] The position information corresponding to the current video frame is predicted based on the motion trend and the position information corresponding to the previous video frame.

[0153] In one embodiment, when the classification and recognition module cannot match the most similar image features, the positioning submodule reads a sequence of location information cached within a recent period of time (a third time interval) from the location information library, and derives data such as the motion trend of the positioning object based on the sequence analysis of the location information, and then predicts the location information of the current video frame based on the motion trend and the location information corresponding to the previous video frame.

[0154] For example, when a user is looking for a store in a large shopping mall, the video screen is briefly blocked. In this case, the user's current location is predicted based on the user's recent location information and movement trajectory, ensuring that the user interface displays real-time positioning information consistently and stably.

[0155] By analyzing previous location information, we can predict the location of the positioned object when it slightly obstructs the image during movement, thereby repairing the problem of temporary loss of location information, ensuring the real-time and consistency of positioning, and improving the user experience.

[0156] As an embodiment of the present application, the indoor positioning system further includes a preprocessing module for performing visualization processing on the video frame to generate a data visual image;

[0157] The identifier is further configured to use the data visual image as auxiliary information, process the initial features and the data visual image, and obtain the target features.

[0158] In one embodiment, a preprocessing module is further provided in the positioning system for performing preliminary processing on the video frame images. Specifically, the preprocessing module performs visualization processing on the video frame images output by the acquisition unit to generate data visual images. The data visual images can be used to assist the identifier in preliminary learning of the video frame images to understand the data features in the video frames. In this embodiment, the identifier uses the data visual images processed by the preprocessing module as auxiliary information, and combines them with the initial features generated by the feature extractor to obtain the target features. In this embodiment, in the process of analyzing the initial features by the identifier, combining the data visual images as auxiliary information helps to improve the accuracy of feature learning and classification and optimize the preprocessing process.

[0159] Figure 3 This is a workflow diagram of the indoor positioning system in one embodiment of the present application. Figure 3As shown, the acquisition unit converts the real-time captured target video into video frame images. The feature extractor in the feature extraction unit extracts convolutional feature maps from the video frames. The extraction submodule in the matching unit extracts partial channels from the convolutional feature maps to remove redundant data. The simplified feature maps are then compressed and aggregated by the feature extractor to obtain a one-dimensional feature vector as the initial features. Furthermore, the preprocessing unit visualizes the video frames to generate data visual images. The recognizer inputs the initial features and the data visual image to generate target features and corresponding types. Based on these types, the classification and recognition module selects sample images of the same type from the storage submodule and identifies them as the images to be matched. The classification and recognition module then controls the calculation module to execute the K-nearest neighbor algorithm, traversing the matching nodes corresponding to each image to be matched in the Kd tree structure index to determine the image features of the sample image that are most similar to the target features. Furthermore, the positioning unit determines the position information of the current video frame based on the most similar image features. Based on these most similar image features, the positioning submodule retrieves the corresponding position information from the position information database and determines it as the position corresponding to the current video frame, thereby completing the real-time positioning of the positioning object. Finally, the positioning submodule sends the current location information to the display module, which displays the location information on the user interface. Furthermore, the positioning submodule sends the current location information to the location information library for caching. The location information library stores location information from the most recent period (the third time interval). Based on the sequence formed by this location information, the positioning submodule can mark the motion trajectory of the positioning object on the map in real time. Furthermore, in the case of occlusion in the video frame image, the positioning submodule can predict the current location of the positioning object through sequence analysis of the location information.

[0160] As an embodiment of the present application, the matching unit further includes an index management module for storing the index of the Kd tree structure and creating mapping information for each sample image; the mapping information includes: the image ID, file name and storage path of the sample image;

[0161] The index management module is also used to scan the storage path of each sample image at a second time interval to detect whether a new file name appears; if a new file name appears, the corresponding old mapping information is searched based on the storage path corresponding to the new file name; the image ID in the old mapping information is obtained, and new mapping information is created in combination with the new file name and the storage path, and the old mapping information is overwritten.

[0162] In this embodiment, the system also includes an index management module that supports the functions of rapid positioning and information update of sample images, thereby enhancing the flexibility and reliability of system positioning. The index management module can be used to cache the index of the Kd tree structure constructed by the calculation module, and to create mapping information for each sample image in the storage submodule. Specifically, the index management module establishes bidirectionally associated mapping information based on the image ID, file name and storage path of the sample image in the storage submodule, as a unique identification mapping for each sample image: Index(f) = (Path, ID), where f is the image file name, Path is the hard disk path, and ID is the image ID (logical identifier). Optionally, the index management module can also establish an index table for recording and standardizing the mapping information of each sample image. By storing the mapping information of the sample image, the storage path, file name, image ID and other information of the sample image can be quickly located. For example, the path of the image img1.jpg is / dataset / images / img1.jpg, and its mapping information is:

[0163] Index(′img1.jpg′)=(′ / dataset / images / img1.jpg′,1001).

[0164] The index management module provides the image retrieval function. During the image retrieval process, the user enters the file name f query After that, the index management module searches and returns the corresponding (Path, ID). The search formula is:

[0165] (Path,ID)=Index(f query ).

[0166] In one embodiment, the index management module also provides a function for updating sample image information. For example, if a user modifies the file name of a sample image in the storage submodule, the old file name f old Change to the new file name f new . The index management module regularly scans the storage submodule according to the second time interval. In actual applications, this can be achieved by scanning the hard disk directory or monitoring the file system events to capture changes. The purpose of this operation is to monitor whether the storage path of each sample image has changed. If a new file name is detected, the corresponding mapping information is searched based on the storage path corresponding to the sample image to obtain the corresponding image ID and the old file name. Further, the image ID, storage path and new file name are bound to establish new mapping information, which is expressed as:

[0167] Index(f new )=(Path, ID).

[0168] In addition, if the mapping information is recorded in the index table, the index table needs to be updated accordingly, and the old mapping information at the corresponding position in the table is overwritten with the new mapping information to complete the update operation of the file name.

[0169] Based on the same inventive concept, an embodiment of the present application provides an indoor positioning method. Figure 4 This is a flow chart of the indoor positioning method proposed in one embodiment of the present application. Figure 4 As shown, the method includes:

[0170] S1: converting the target video collected in real time into video frames according to a first time interval;

[0171] S2: extracting initial features from the current video frame using a pre-trained convolutional neural network; the initial features are convolutional feature maps or feature vectors;

[0172] S3: Processing the initial features to obtain target features;

[0173] S4: constructing an index of a Kd tree structure based on the image features of the sample image; each node in the index stores the image features of a sample image; the closer the distance between nodes in the index, the higher the similarity; the sample image contains at least one of the following detection objects: indoor buildings, indoor objects, and indoor signs;

[0174] S5: using a k-nearest neighbor algorithm to match the image feature most similar to the target feature in the index;

[0175] S6: Acquire corresponding position information based on the most similar image feature, and locate the current video frame based on the position information.

[0176] As an embodiment of the present application, initial features are extracted from the current video frame by using a pre-trained convolutional neural network, including:

[0177] Processing the video frame through a convolutional neural network to generate a convolution feature map, and determining the convolution feature map as the initial feature; or

[0178] A random sampling algorithm is used to extract feature data of some channels from the convolution feature map to generate a simplified feature map; the simplified feature map is aggregated into a feature vector, and the feature vector is determined as the initial feature.

[0179] As an implementation of the present application, a k-nearest neighbor algorithm is used to match the image feature most similar to the target feature in the index, including:

[0180] Determine the image to be matched based on all sample images;

[0181] Traversing the to-be-matched nodes where the image features of each to-be-matched picture are located based on the index, and calculating the Euclidean distance between the image features of each to-be-matched node and the target feature;

[0182] The image feature corresponding to the smallest Euclidean distance that is not greater than the distance threshold is screened out and determined as the most similar image feature.

[0183] As an implementation of the present application, determining the image to be matched based on all sample images specifically includes:

[0184] Analyzing the initial features to obtain corresponding target types;

[0185] Based on the target type, sample images of the same type are used as images to be matched;

[0186] Each node in the index stores the picture ID corresponding to the image feature;

[0187] Traversing the to-be-matched nodes where the image features of each to-be-matched image are located based on the index, and calculating the Euclidean distance between the image features of each to-be-matched node and the target feature, specifically includes:

[0188] Based on the picture IDs corresponding to all the pictures to be matched, search the index to determine the location of the corresponding node to be matched;

[0189] Based on the position of each to-be-matched node, the Euclidean distance between the image feature of each to-be-matched node and the target feature is calculated by traversing the layers from top to bottom.

[0190] As an implementation of the present application, the image feature corresponding to the smallest Euclidean distance not greater than the distance threshold is screened out and determined as the most similar image feature, including:

[0191] Compare the Euclidean distance of each node to be matched with the distance threshold. If the Euclidean distance of the current node is less than or equal to the distance threshold, add the image ID stored in the current node to the result set. If the Euclidean distance of the current node is greater than the Euclidean distance in the result set, skip traversing the child nodes of the node.

[0192] Based on the node corresponding to the picture ID in the result set, further traverse the child nodes in the next layer of the node; if the Euclidean distance of the child node is less than or equal to the Euclidean distance corresponding to the picture ID in the result set, then add the picture ID corresponding to the child node to the result set and overwrite the old picture ID;

[0193] After traversing all nodes to be matched, if the result set is empty, it is determined that there is no image feature similar to the target feature; if the result set is not empty, the image feature corresponding to the picture ID currently stored in the result set is determined as the most similar image feature.

[0194] As an implementation manner of the present application, the indoor positioning method further includes:

[0195] After locating the current video frame, the corresponding position information is cached and displayed on the user interface;

[0196] Obtain all location information cached within a third time interval from the current time, mark the real-time movement trajectory on the map based on the order of all location information, and display the map in real time on the user interface; the third time interval is at least twice as long as the first time interval.

[0197] As an implementation manner of the present application, the indoor positioning method further includes:

[0198] In the absence of a sample picture similar to the video frame, obtaining all position information cached within a third time interval from the current time, and analyzing all the position information to obtain a motion trend;

[0199] The position information corresponding to the current video frame is predicted based on the motion trend and the position information corresponding to the previous video frame.

[0200] As an implementation manner of the present application, before processing the initial features, the method further includes:

[0201] Performing visualization processing on the video frames to generate data visual images;

[0202] The initial features are processed, specifically comprising: using the data visual image as auxiliary information, processing the initial features and the data visual image, and obtaining the target features.

[0203] As an embodiment of the present application, before extracting initial features from the current video frame using a pre-trained convolutional neural network, the method further includes:

[0204] Acquire multiple images containing detection objects as sample images; the detection objects include at least: indoor buildings, indoor objects, and indoor signs;

[0205] Add corresponding image ID and type label to each sample image, and store all sample images;

[0206] Based on the sample images, a convolutional neural network is trained using a transfer learning method, and the fully connected layer is removed from the trained convolutional neural network, and a feature extractor is constructed based on the remaining network;

[0207] The image features of each sample picture are extracted by the feature extractor and stored corresponding to the sample picture.

[0208] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0209] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0210] For the sake of simplicity, the method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and components involved are not necessarily required by this application.

[0211] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0212] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0213] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0214] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0215] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the underlying inventive concepts. Therefore, this application is intended to include the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0216] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0217] The above is a detailed introduction to the indoor positioning system and method provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An indoor positioning system, characterized in that: include: An acquisition unit, configured to convert the target video acquired in real time into video frames according to a first time interval; A feature extraction unit, comprising a feature extractor and a recognizer; the feature extractor is used to extract initial features from the current video frame using a pre-trained convolutional neural network; The initial feature is a convolution feature map or a feature vector; the identifier is used to process the initial feature to obtain the target feature; A calculation module is configured to construct a Kd tree structure index based on the image features of the sample images; each node in the index stores the image features of a sample image; the closer the nodes in the index, the higher the similarity; and the sample images contain at least one of the following detection objects: indoor buildings, indoor objects, and indoor signs; The matching unit includes a classification and recognition module and a storage submodule; the storage submodule is used to store sample images and corresponding image features; the classification and recognition module is used to control the calculation module to execute the K-nearest neighbor algorithm to match the image feature most similar to the target feature in the index; The positioning unit includes a positioning submodule and a position information library; the position information library is used to store feature data and corresponding position information; The positioning submodule is configured to obtain corresponding position information from the position information library based on the most similar image features, and to position the current video frame based on the position information.

2. The indoor positioning system according to claim 1, characterized in that The feature extractor is specifically configured to process the video frame through a convolutional neural network to generate a convolutional feature map; The matching unit further includes an extraction submodule for extracting feature data of some channels from the convolution feature map using a random sampling algorithm to generate a simplified feature map; The feature extractor is further configured to aggregate the simplified feature map into a feature vector; The identifier is specifically used to process the convolution feature map or the feature vector as an initial feature to obtain the target feature.

3. The indoor positioning system according to claim 1, characterized in that The classification and recognition module is used to control the calculation module to execute the K-nearest neighbor algorithm to match the image feature most similar to the target feature in the index, specifically including: Determine the image to be matched based on all sample images; Controlling the calculation module to traverse the to-be-matched nodes where the image features of each to-be-matched picture are located based on the index, and calculating the Euclidean distance between the image features of each to-be-matched node and the target feature; The image feature corresponding to the smallest Euclidean distance that is not greater than the distance threshold is screened out and determined as the most similar image feature.

4. The indoor positioning system according to claim 3, characterized in that The identifier is further configured to analyze the initial features to obtain a corresponding target type; The storage submodule is further used to store the type of each sample image; The classification and recognition module is configured to determine the to-be-matched pictures based on all the sample pictures, specifically including: based on the target type, taking the sample pictures of the same type as the to-be-matched pictures; Each node in the index stores the picture ID corresponding to the image feature; The calculation module is used to traverse the to-be-matched nodes where the image features of each to-be-matched picture are located based on the index, and calculate the Euclidean distance between the image features of each to-be-matched node and the target feature, specifically including: Based on the picture IDs corresponding to all the pictures to be matched, search the index to determine the location of the corresponding node to be matched; Based on the position of each to-be-matched node, the Euclidean distance between the image feature of each to-be-matched node and the target feature is calculated by traversing the layers from top to bottom.

5. The indoor positioning system according to claim 3 or 4, characterized in that: The classification and recognition module is used to screen out image features corresponding to the smallest Euclidean distance not greater than the distance threshold and determine them as the most similar image features, specifically including: Compare the Euclidean distance of each node to be matched with the distance threshold. If the Euclidean distance of the current node is less than or equal to the distance threshold, add the image ID stored in the current node to the result set. If the Euclidean distance of the current node is greater than the Euclidean distance in the result set, skip traversing the child nodes of the node. Based on the node corresponding to the picture ID in the result set, further traverse the child nodes in the next layer of the node; if the Euclidean distance of the child node is less than or equal to the Euclidean distance corresponding to the picture ID in the result set, then add the picture ID corresponding to the child node to the result set and overwrite the old picture ID; After traversing all nodes to be matched, if the result set is empty, it is determined that there is no image feature similar to the target feature; if the result set is not empty, the image feature corresponding to the picture ID currently stored in the result set is determined as the most similar image feature.

6. The indoor positioning system according to claim 1, characterized in that The position information library is further used to cache corresponding position information after positioning the current video frame; The system further includes a display module for displaying position information corresponding to the current video frame on a user interface; The positioning submodule is further configured to obtain all cached location information within a third time interval from the current time, and mark the real-time motion trajectory on the map based on the order of all the location information; the third time interval is at least twice as long as the first time interval; The display module is also used to display the map in real time on the user interface.

7. The indoor positioning system according to claim 6, characterized in that: The positioning submodule is further configured to obtain, when there is no sample image similar to the video frame, all cached position information within a third time interval from the current time, and obtain a motion trend based on analysis of all the position information; The position information corresponding to the current video frame is predicted based on the motion trend and the position information corresponding to the previous video frame.

8. The indoor positioning system according to claim 1, wherein: It also includes a pre-processing module for performing visualization processing on the video frames to generate data visual images; The identifier is further configured to use the data visual image as auxiliary information, process the initial features and the data visual image, and obtain the target features.

9. The indoor positioning system according to claim 1, characterized in that The matching unit further includes an index management module for storing the index of the Kd tree structure and creating mapping information for each sample image; the mapping information includes: the image ID, file name and storage path of the sample image; The index management module is also used to scan the storage path of each sample image at a second time interval to detect whether a new file name appears; if a new file name appears, the corresponding old mapping information is searched based on the storage path corresponding to the new file name; the image ID in the old mapping information is obtained, and new mapping information is created in combination with the new file name and the storage path, and the old mapping information is overwritten.

10. An indoor positioning method, characterized in that: The indoor positioning system according to any one of claims 1 to 9 comprises: Converting the target video captured in real time into video frames according to a first time interval; Extracting initial features from the current video frame using a pre-trained convolutional neural network; the initial features are convolutional feature maps or feature vectors; Processing the initial features to obtain target features; Constructing a Kd tree structure index based on the image features of the sample images; each node in the index stores the image features of a sample image; the closer the nodes in the index are, the higher the similarity; the sample images contain at least one of the following detection objects: indoor buildings, indoor objects, and indoor signs; Using a k-nearest neighbor algorithm, matching the image feature that is most similar to the target feature in the index; Corresponding position information is acquired based on the most similar image feature, and the current video frame is positioned based on the position information.

11. The indoor positioning method according to claim 10, characterized in that: Before extracting initial features from the current video frame through a pre-trained convolutional neural network, it also includes: Acquire multiple images containing detection objects as sample images; the detection objects include at least: indoor buildings, indoor objects, and indoor signs; Add corresponding image ID and type label to each sample image, and store all sample images; Based on the sample images, a convolutional neural network is trained using a transfer learning method, and the fully connected layer is removed from the trained convolutional neural network, and a feature extractor is constructed based on the remaining network; The image features of each sample picture are extracted by the feature extractor and stored corresponding to the sample picture.

Images